I think in this case it's more like you've bought a truck but you put a lawnmower engine in it.
It's a big step between esoterics and "please don't use the cheap shitty DAC that was integrated into your monitor to satisfy a feature checklist".
The amp part will matter a lot the more impedance the cans have, and many of the better headphones are high(er)-impedance. Might as well spring for a headphone amp + DAC combo just to see how it sounds. It's not that expensive, FiiO has some for under 80 bucks, for example.
A man sits on a weight bench, wondering if the equipment is defective because he can't seem to bench press 800 lb...