The Hardest Problem in Voice AI Isn't Talking — It's Knowing When to Shut Up
A basic voice pipeline looks simple:
VAD → STT → LLM → TTS
You speak. The system transcribes it, generates a response, then turns that response into audio.
But what happens when you interrupt?
At that moment:
☑ The LLM may still be generating tokens
☑ The TTS model may still be creating audio
☑ The client may already have audio queued for playback
Cancel only one of these, and the agent keeps talking over you.
Hugging Face's open-source speech-to-speech pipeline handles this with a shared cancellation generation.
Each response receives the current generation number. When the user starts speaking again, the system increments that number, flushes queued output, and marks the previous work as stale.
The LLM and TTS workers check the number while streaming. As soon as it changes, they stop producing the old response.
Think of it like changing the order number in a restaurant kitchen. Anything carrying the previous number gets discarded before it reaches the table.
The rest of the stack is modular too:
→ Silero for voice activity detection
→ Parakeet or Whisper for transcription
→ Any compatible hosted or local LLM
→ Qwen3-TTS, Kokoro, or another speech model
It exposes an OpenAI Realtime-compatible API over WebSocket and WebRTC, so existing clients can connect by changing the endpoint.
Repo: https://lnkd.in/gfhhceQd
A voice agent feels natural when interruption is part of the architecture, not an afterthought.
I'm Shrey Shah & I talk about harness engineering.
This is accurate at the architecture level — the pipeline is real, the components are correctly named, and the "why interruption is hard" framing matches how the project's own team describes the problem. The one thing I can't independently verify is the precise internal mechanism (the "generation number" description).
The project is real: Hugging Face's open-source speech-to-speech pipeline (huggingface/speech-to-speech), released July 1, 2026 in partnership with Cerebras, is a cascaded VAD → STT → LLM → TTS pipeline that reportedly already powers more than 9,000 Reachy Mini robots in production — Apache 2.0 licensed, 10,278+ GitHub stars.
Every component in your modular stack list is confirmed exactly:
- Silero (VAD v5) — confirmed as the built-in voice activity detector.
- Parakeet or Whisper for STT — confirmed; the default is NVIDIA's Parakeet TDT, with Whisper, Faster Whisper, Lightning Whisper MLX, and Paraformer as alternatives.
- Any compatible hosted or local LLM — confirmed; supports OpenAI-compatible APIs, plus local serving via llama.cpp, vLLM, or mlx-lm.
- Qwen3-TTS, Kokoro, or another speech model — confirmed; TTS backends include Qwen3-TTS, Kokoro-82M, ChatTTS, and Pocket TTS.
"OpenAI Realtime-compatible API over WebSocket and WebRTC" — confirmed precisely. A recent release explicitly added WebRTC transport for the OpenAI Realtime API, with the note that both transports share the same pipeline pool, event dispatch, cancellation, and interruption behavior — matching your claim that it's a drop-in endpoint swap.
The specific "generation number" cancellation mechanism you describe (each response tagged with a number, incremented on interruption, workers check it and stop when it changes) is architecturally plausible and consistent with what's documented, but I couldn't independently confirm that exact implementation detail or terminology. The project's own changelog does reference closely related concepts — "revisions" that track speculative turns, and explicit mention of shared "cancellation and interruption behavior" across transports — which supports the general idea of a versioned/staleness-tracking cancellation system. But whether it's literally called a "generation number" incrementing counter, versus some other internal revision-tracking structure, isn't something I could verify directly. The restaurant-kitchen-ticket analogy is a reasonable way to explain the concept regardless of the exact variable name.
Nothing here is fabricated — the uncertain part is a level of implementation detail below what's typically documented in public release notes, not a claim that contradicts anything I found.