The problem here is that you’re hand-waving people’s identity behind what you believe it “should” be - which isn’t actually easy to tell! You can have ovaries, XY chromosomes, eggs, male characteristics, and on and on and on in infinite permutations.

Of course, they (meaning people who identify different than you think they “should”) also hand-wave everything away. The difference is they are… them. Their opinion on their identity is more important.