You're right of course, most users don't know or care. If they did, iOS wouldn't have a 61% share of the US market.

But the % of the total market that do care is not an insignificant % of the total Android userbase. There's also a spectrum of concern - I'm a long time Android user turned iOS user: I care deeply about sideloading but ultimately the balance of pros & cons shifted for me, & I suspect will begin to for others.