Yes, but SSL connections are fine once they get going -- the nasty part is how many round-trips are needed to complete the handshake. Any latency between the client and the server is going to be multiplied several times over as they do the initial ritual of verifying public keys and establishing a session key.

The trick here is to cut down on the latency of establishing the session.