As I understand my favorite resolver - unbound - has separate "infra" cache that also caches RTT times, and "down" status of name servers:

https://nlnetlabs.nl/documentation/unbound/unbound.conf/#inf...