This is not strictly speech-to-speech, but I quite like it when working with Claude Code or other CLI Agents:
STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps confirm understanding as well as likely helps the CLI agent stay on track.
TTS: Pocket-TTS [2], just 100M params, and amazing speech quality (English only).
I made a voice plugin [3] based on this, for Claude Code so it can speak out short updates whenever CC stops. It uses a non-blocking stop hook that calls a headless agent to create the 1/2-sentence summary. Turns out to be surprisingly useful. It's also fun as you can customize the speaking style and mirror your vibe etc.
The voice plugin gives commands to control it:
/voice:speak stop
/voice:speak azelma (change the voice)
/voice:speak <your arbitrary prompt to control the style or other aspects>
Hex is my new favorite STT on MacOS. Also uses Parakeet V3. I didn't think it could possibly be faster than Handy, but it is much faster - even long ramblings transcribed within a second. It's MacOS only, leverages the CoreML / Apple Neural Engine.
For local speech-to-text, Whisper remains the gold standard - you can run it locally with good accuracy across languages. For speech-to-speech, you'd typically chain Whisper with a local TTS model like Coqui TTS or use something like Tortoise TTS for higher quality but slower processing. The key is balancing accuracy, speed, and resource usage based on your specific use case. If you're doing content creation workflows, consider what post-processing you might need - sometimes the raw transcription needs structure and enhancement beyond just accurate words.
+1 on the post-processing point. Raw Whisper output is ~90% there but punctuation, grammar, and formatting are the missing piece.
I built MumbleFlow to address exactly this — whisper.cpp for STT plus llama.cpp for smart text cleanup, all running on-device. Metal/CUDA accelerated, sub-second latency on Apple Silicon. Global hotkey works in any app.
Nice, I’ll have to try it out. They should really make a uv-installable CLI tool like pocket-TTS did. People underestimate just how much more immediately usable something becomes when you can simply get something by doing “uv tool install …”
Hi, so I'm looking for an stt that can happen on a server/cron, that will use a small local model (I have 4 vCPU threadripper CPU only and 20G ram on the server) and be able to transcribe from remote audio URLs (preferably, but I know that local models probably don't have this feature so will have to do something like curl the audio down to memory or /tmp and then transcribe and then remove the file etc).
STT: Handy [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps confirm understanding as well as likely helps the CLI agent stay on track.
TTS: Pocket-TTS [2], just 100M params, and amazing speech quality (English only). I made a voice plugin [3] based on this, for Claude Code so it can speak out short updates whenever CC stops. It uses a non-blocking stop hook that calls a headless agent to create the 1/2-sentence summary. Turns out to be surprisingly useful. It's also fun as you can customize the speaking style and mirror your vibe etc.
The voice plugin gives commands to control it:
[1] Handy https://github.com/cjpais/Handy[2] Pocket-TTS https://github.com/kyutai-labs/pocket-tts
[3] Voice plugin for Claude Code: https://github.com/pchalasani/claude-code-tools?tab=readme-o...