Remember that all motion is relative. You can’t model space travel as simply going from point to point. Instead, you transfer from orbit to orbit and have to match velocities wherever you go. Relative to the propellant depot you have to reach zero velocity, but the propellant depot will be orbiting something at a very high speed just like you are.

You will likely have to go out of your way somewhat to reach a propellant depot, so you don’t want your propellant depots at the bottom of heavy gravity wells or in hard-to-reach orbits. But it’s still a viable strategy.