For me the mouse is never quicker. I think that over time your brain kind of memorizes the number of lines (rows) and characters (columns) available in your screen and it certainly gets to a point where you don't really think about the science behind motions and you end up just using motions defined through past experiences. I amaze myself (and my coworkers) when I apply a seemingly random number to a motion and I end up exactly where I wanted to be. Last time a coworker asked me if I'm a math freak and I did the calculations in my head when doing a complex motion in a second, and even though I answer that I am (jokingly), I know that it's actually just a combination of muscle memory and past memories.
In any case, even though I don't agree with the school that says that to learn vi/vim you should forgo arrow keys and non vim (cmd + s to save, command + left arrow to go the left of the line, etc) key combinations, I think that a major point of inflection while learning the editor is choosing a trick/combination/command that you want to learn because it will increase your productivity and keep forcing it, even if it means not using the arrow keys or other combinations you're not used to, until it becomes second nature. And then do the next one and so on.
In any case, even though I don't agree with the school that says that to learn vi/vim you should forgo arrow keys and non vim (cmd + s to save, command + left arrow to go the left of the line, etc) key combinations, I think that a major point of inflection while learning the editor is choosing a trick/combination/command that you want to learn because it will increase your productivity and keep forcing it, even if it means not using the arrow keys or other combinations you're not used to, until it becomes second nature. And then do the next one and so on.