One time, many moons ago now, I noticed Pulse taking a reasonable fraction of a core and thought I'd zap PA to try to use less CPU.
Unfortunately, using my media player without PA actually used more CPU overall -- the audio player's CPU use went up by more than PA had been using. So I went back to using PA.
There is a sibling comment describing the same experience. But I think that this behavior can't be reproduced nowadays.
I think it must depend on the relative performance of the software resamplers involved (ALSA's in-kernel plughw vs. PulseAudio's vs. your media player's).
Do you remember which media player triggered this?
Unfortunately, using my media player without PA actually used more CPU overall -- the audio player's CPU use went up by more than PA had been using. So I went back to using PA.