voice activity detection belongs in the turn-taking protocol

A browser voice interface needs separate boundaries for detected speech, committed text, quiet time, and model interruption.

A speech recognizer returns text. A voice interface must also decide when the user has finished a turn.

I hit this boundary while building a browser-local voice prototype. The interface used MoonshineJS for speech recognition and WebLLM for local responses. Background sound kept the recognizer active after I stopped speaking. A short transcript such as hello could also disappear before the model answered.

The language model was not the cause. The turn controller treated every speech event as confirmed user input.

speech recognition is not turn detection

Automatic speech recognition (ASR) converts audio into text. Voice activity detection (VAD) estimates whether an audio frame contains speech.

The first version created MicrophoneTranscriber with its third argument set to false. In MoonshineJS, that value disables VAD mode and enables frequent streaming updates. The MoonshineJS microphone guide documents this mode for live transcript updates.

Streaming updates were useful for captions. They were a poor turn boundary in a room with background sound. The recognizer could keep revising the same utterance because it had no strong speech-end event.

The immediate fix was small:

const transcriber = new MicrophoneTranscriber(
  "model/tiny",
  callbacks,
  true,
);

This enables the VAD path. MoonshineJS then commits text after a speech event instead of relying on continuous transcript updates alone.

a speech-start event is tentative

Enabling VAD fixed only half of the problem. The interface still cleared the previous exchange inside onSpeechStart.

That transition was too eager. A speech-start event means the detector found speech-like audio. It does not mean the recognizer has valid text. A false start could erase hello even when no new transcript followed.

I split the lifecycle into tentative and confirmed transitions. The controller now uses these rules:

The important change is where destructive cleanup happens. onSpeechStart no longer clears the committed exchange. The first non-empty transcript confirms that a new turn exists.

const confirmSpeechTurn = () => {
  if (speechTurnHasTextRef.current) return;
  speechTurnHasTextRef.current = true;
  if (generationAbortRef.current) interruptGeneration();
  setMessages([]);
  setAssistantDraft("");
};

Empty transcripts also return early. Background audio can change the listening indicator, but it cannot delete text or start generation.

a quiet window separates commits from responses

VAD produces a useful speech boundary, but people pause inside sentences. Starting a response on every commit makes the model interrupt too early.

The controller waits 720 milliseconds after a final transcript. If another final arrives during that window, it joins the text and restarts the timer.

clearTimeout(quietTimer);
quietTimer = window.setTimeout(() => {
  generate(committedText);
}, 720);

This delay is a product value, not a model constant. A command interface can use a shorter delay. Dictation needs a longer one. The controller owns the value because it defines conversation behavior.

The response path uses an AbortController. When confirmed speech arrives during generation, the controller aborts the current request and tells the WebLLM worker to interrupt. The user can take the floor without waiting for the model to finish.

a fallback can change the privacy boundary

I added a Web Speech API mode to compare recognition quality. I did not make it the silent default.

Browser speech recognition does not always run on the device. The processLocally property requests local recognition, but the browser also needs a compatible language pack. MDN documents the related available() and install() checks.

On the tested Brave installation, local en-US recognition was unavailable. The browser mode could therefore use a remote recognition service. That is a different privacy contract from the default MoonshineJS path.

An engine adapter can normalize callbacks, but it cannot make those contracts equivalent. The interface must disclose the difference or keep the remote path behind an explicit option.

test the turn protocol

The final checks covered more than recognition accuracy. Browser smoke tests verified that empty results do not start a turn, the active exchange remains visible after a false start, and a new turn can interrupt generation. The tests also checked the setup states and viewport overflow.

npm run check and the browser smoke suite passed after the change.

Treat audio events as evidence, not commands. Confirm text before you replace committed UI state. The same controller should own the quiet window and response interruption. That design remains stable when the speech engine changes.