I would second @ASIO ‘s recommendation. By using only 1 pin, you are throwing away half the performance of the DAC. Finding a good transformer based solution to convert the signal is essential. It may be worth looking for a used set or asking your dealer if they can get a discounted set. It would be well worth it at least to borrow a set for testing purposes and see how much it helps. I know second hand sets come up from time to time.
I have a fully balanced, tubed, PreAmp. To use single ended output from my preamp, I have to flip a switch, which grounds out the cold pin, to prevent hum. I was running balanced out to my monoblocks, and it seemed fine. But, I noticed, on a picture of the inside of my amp, that it is not running fully balanced, also grounding the cold pin.
It got me thinking. Many of the amps I want to try, and/or build are single ended only. So, I dug out my Jensen XLR->RCA converter, and hooked up my current amps via RCA, instead of XLR, and I swear it sounds better. This converter stays in the system for good now, unless I get a truely balanced amp. (The DIY Aleph J was the only truely balanced amp I have used)