The most fundamental element of science is not today's body of understanding, but the process with which we arrived at that knowledge. A process that generally recognizes human fallibility and how easy it is for us to trick ourselves. So yes, a lot of the stuff we think we know today might be wrong, but the point of science is continuous refinement and trying to eliminate the biases you mentioned.

Applying the process of science to astrology causes the whole edifice to quickly crumble[1], that's why it can be dismissed out of hand. Not just because of what we think we know about the universe.

[1] https://www.youtube.com/watch?v=rYm_0AGMHIg