it isn't. It is not about the fact that you hear something but about the fact that your brain focuses on something different. Listening to someone talking is worse than listening to music in this case. I want to see you play AoE on the same level while actively listening to a podcast or something.
I am not saying you shouldn't listen to anything. There are differences between drivers anyway and there will always be people who put others in danger without listening to anything. You should be aware of the fact that it takes up your concentration.
For me it doesn't take up more concentration than simply thinking about stuff. Either way I'm not focusing on driving, it is almost all autopilot especially on highways.