I used to develop for the N64 and I can confirm that it is true. It is crazy how much faster the CPU is compared with not-in-cache RAM access.

Optimizing for RAM access instead of CPU instruction speed can make your code magnitudes faster.