Before ASR, Before the LLM, Before TTS: Why Voice Activity Detection Comes First

I trained a VAD model from scratch in about a 20-hour A100 GPU run.
But first question: what is VAD? Voice Activity Detection asks one simple thing: is anyone speaking right now?
𝐖𝐡𝐚𝐭 𝐕𝐀𝐃 𝐃𝐨𝐞𝐬
- VAD reads audio frame by frame.
- Each frame is labeled as speech or non-speech.
- Silence and noise should stay inactive.
- Human speech should switch listening on.
[This type of post takes a lot of effort to make, so I would appreciate it if you could repost this so that more people can learn about Audio AI fundamentals.]
𝐖𝐡𝐲 𝐈𝐭 𝐌𝐚𝐭𝐭𝐞𝐫𝐬
- Classic agents run VAD before ASR, the LLM, and TTS.
- VAD gives boundaries: when to listen, stop, and ignore the room.
- If VAD is wrong, the rest of the voice stack already feels broken.
𝐌𝐨𝐝𝐞𝐫𝐧 𝐒𝐩𝐞𝐞𝐜𝐡-𝐓𝐨-𝐒𝐩𝐞𝐞𝐜𝐡
- Newer systems are moving beyond strict turn-taking.
- Full-duplex models listen while speaking.
- That enables barge-in, interruptions, pauses, and natural timing.
- The timing problem becomes turn detection, semantic VAD, and interruption control.
𝐅𝐫𝐚𝐦𝐞𝐬 𝐀𝐧𝐝 𝐇𝐨𝐩 𝐒𝐞𝐪𝐮𝐞𝐧𝐜𝐞
- The waveform is split into 25 ms frames.
- The model advances with a 10 ms hop, so nearby frames overlap.
- Speech is continuous, but models need fixed slices.
- Overlap keeps context between frames.
𝐒𝐜𝐨𝐫𝐞𝐬 𝐓𝐨 𝐃𝐞𝐜𝐢𝐬𝐢𝐨𝐧𝐬
- Every 10 ms frame gets P(speech), a value between 0 and 1.
- Above the start threshold, speech begins.
- Below the end threshold for long enough, the segment closes.
- Those rules turn probabilities into speech regions.
𝐌𝐨𝐝𝐞𝐥 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞
- I used a MarbleNet-style network from scratch.
- It uses stacked 1D depthwise-separable convolution blocks.
- Blocks use residual connections, batch norm, ReLU, and dropout.
- A final 1x1 convolution maps features to speech/non-speech logits.
𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐒𝐞𝐭𝐮𝐩
- Trained in roughly 20 GPU-hours.
- The model has about 143k parameters.
- The causal receptive field is around 1.7 seconds.
- Training used LibriSpeech, MUSAN, and RIRS mixtures.
- Checkpoint exported to ONNX.
𝐓𝐞𝐬𝐭 𝐑𝐞𝐬𝐮𝐥𝐭𝐬
- Clean speech was detected as speech.
- Generated silence returned no speech segments.
- Speech mixed with white noise still produced a usable region.
[Just a quick reminder to repost this post]
Paper Source
https://lnkd.in/gMBcPP6m
Here is the Github Code
https://lnkd.in/gZygfWQ8
Model weights
https://lnkd.in/g9Kmrh2e
I have also implemented a Speech to text transformer from scratch
Here is a Detailed Visual Blog & Video
https://lnkd.in/gs8bGSwW
I Mayank Pratap Singh will be posting more Audio AI implementations and breaking down Audio and LLM inference concepts in future posts, so if you are interested in Audio deep learning and LLM inference optimization, follow me Mayank Pratap Singh for that.

Technically accurate throughout — this reads like a well-informed, correct summary of standard VAD concepts and NVIDIA's MarbleNet architecture.

• "VAD reads audio frame by frame, labels speech/non-speech" — accurate, standard definition of Voice Activity Detection as a binary framewise classification task.
• "25 ms frames, 10 ms hop, overlap keeps context" — accurate. This is the standard convention in speech processing (also used for MFCC/spectrogram extraction generally), so the framing is technically correct and not something unique to this project.
• "P(speech) per frame, start/end thresholds turn probabilities into speech regions" — accurate description of how threshold-based VAD post-processing typically works (onset/offset hysteresis to avoid flickering on/off at the boundary).
• MarbleNet architecture: stacked 1D depthwise-separable (time-channel separable) convolution blocks, residual connections, batch norm, ReLU, dropout, final 1x1 conv to logits — accurate and precisely matches the actual MarbleNet paper's described architecture (Jia, Majumdar, Ginsburg, NVIDIA, ICASSP 2021). MarbleNet is a real, published, well-known lightweight VAD architecture, and this description of it is correct down to the details.
• "Modern full-duplex speech-to-speech systems moving beyond strict turn-taking, semantic VAD, interruption control" — accurate characterization of where the field has moved; this reflects real, current terminology and direction in voice AI research (turn detection and semantic VAD are genuinely active problem areas distinct from classic energy/frame-level VAD).
• Personal project claims (20-hour A100 run, ~143k parameters, ~1.7s receptive field, trained on LibriSpeech/MUSAN/RIRS, exported to ONNX, test results) — these are first-person claims about the poster's own work that I can't independently verify, but they're internally consistent and plausible: MarbleNet-style models are known for being extremely lightweight (the original paper touts ~1/10th the parameters of prior SOTA), so a sub-150k-parameter model trained in ~20 GPU-hours is a believable scale for this architecture. LibriSpeech (clean speech), MUSAN (noise), and RIRs (room impulse responses for reverb augmentation) are indeed the standard dataset combination used for training robust VAD/ASR front-ends.