Yes, I think you are right. The headphone output is shot. But I don't think that M-Audio USB AI are crap. On the contrary- most other are crap. And why should Avid choose M-Audio as a partner?
And yes, the right solution is to attach a headphone amplifier to the secondary Line Output.
Most of the USB Stereo AI I have had my hands on have had all sorts of issues. I think the best and most reliable are M-Audio, Focusrite, Steinberg, Presonus, TASCAM, Audient. My next AI will be M-Audio again!