I've also asked about headphone swapping and two things I learned from responses were....
If you have headphones that have volume control, it allows you to overcome the issue you're experiencing.
It may have something to do with impedance rating.
I did a quick search. This is one answer that may explain the issue....
"The power required to operate headphones is a function of the driver size and mass, whether the phones are open, closed, or in-ear, etc. The impedance of headphones is usually designed to get an acceptable volume level (power = voltage^2 / impedance) at the voltage level expected on the driving equipment. It's not necessarily a straightforward relationship of low impedance = cheap, high-impedance = pro.
Many really fine headphones have large drivers (more power), are open-air (more power), AND expect to be run from higher-voltage professional/amplified equipment... which sort of cancels out the impedance question from a design perspective. Apple earbuds are 23 ohms... my fave studio headphones, Audio-Technica ATH-M50s, are 38 ohms... one of the most common studio headphones, Sony MDR7506, are 63 ohms. So, more "pro" headphones are higher impedance, but that just means that you will get less power (lower volume) on a low-voltage audio source.
There are other good reasons to use an external amplifier... RF/EMI noise isolation, etc... but I think the main reason you might want to with better headphones is that the volume may be low with higher-impedance headphones."