Don’t Build A Second Product For Voice

Most voice demos have no saved conversations, permissions, model settings, memory, billing history, or old transcripts. They can stop when the spoken response plays. RecallMEM could not.

I hit this while adding Deepgram Voice Agent to RecallMEM, my local-first AI memory app. RecallMEM already had Postgres, pgvector, saved conversations, provider settings, and memory extraction.

Speech-to-text was only the live transport: record audio, transcribe it, send text to a model, and play speech back.

The voice path needed to use the same memory, transcripts, settings, tools, authorization boundaries, and user context as text chat.

The wrong architecture looks reasonable:

Browser mic
  -> audio blob
  -> app server
  -> speech-to-text
  -> LLM
  -> text-to-speech
  -> browser speaker

This flow works for a one-off bot. In RecallMEM, it would have created a second path around state the application already owned.

The browser owns the microphone, but it should not own memory or credentials. The model speaks, but it should not choose its own authorization context. The new session still belongs in the existing transcript.

RecallMEM uses this shape:

Browser mic
  -> Deepgram Voice Agent
       -> listens
       -> handles turns
       -> speaks back
       -> asks for tools when needed
  -> app-owned tool routes
       -> memory
       -> transcripts
       -> settings
       -> auth boundaries
  -> normal app persistence

Deepgram handles listening, turn-taking, tool calls, and speech. RecallMEM decides what the agent can read, what tools it can call, what gets stored, and where the transcript goes.

The model talks. RecallMEM owns the data.

Voice feels client-side because the mic is client-side. That does not mean the browser gets to own the important parts.

In RecallMEM, the browser asks the app server for the Voice Agent config. The server keeps the long-lived Deepgram API key. It creates a short-lived browser token, builds the settings payload, and sends back only what the browser needs to start the session.

The server builds that configuration from user settings, available providers, memory rules, selected models, and tool access. Keeping those decisions server-side avoids duplicating them in browser code.

In practice, the browser captures and plays audio while the server builds the product configuration and keeps long-lived credentials.

Deepgram Voice Agent is not “STT plus TTS.” The Settings message is where the live system gets assembled.

In RecallMEM, the settings define the audio format, listening model, thinking model, speaking voice, tools, and greeting.

const settings = {
  type: "Settings",
  audio: {
    input: { encoding: "linear16", sample_rate: 16000 },
    output: { encoding: "linear16", sample_rate: 48000, container: "none" },
  },
  agent: {
    listen: {
      provider: {
        type: "deepgram",
        version: "v2",
        model: "flux-general-en",
        keyterms,
      },
    },
    think: thinkChain,
    speak: speakChain,
    greeting: "Hey, I'm here. What's up?",
  },
};

Using Flux required more than changing the model name. The listen provider also needed version: "v2", and the payload could not carry Nova-style options such as smart_format.

RecallMEM also disables slow local models for live voice. Local Gemma/Ollama can still work for text chat. They do not belong in the live voice path unless the user enjoys dead air.

I disabled slow local models for live voice because their pauses felt like a broken connection. The same delay was tolerable in text chat.

Realtime bugs have a special talent for making you feel stupid. The WebSocket can be open. The mic can be ready. The agent can still not be ready for your audio yet.

RecallMEM handles Welcome, sends Settings when the socket opens, and only streams microphone audio after SettingsApplied. That last gate matters more than the event order.

Browser mic
  -> asks app server for a short-lived Deepgram token
  -> opens Deepgram Voice Agent WebSocket
  -> sends Settings on socket open
  -> handles Welcome
  -> waits for SettingsApplied
  -> streams microphone audio

RecallMEM manages the WebSocket directly and uses Deepgram’s browser helpers for microphone capture and PCM playback. The microphone callback checks both configuration and socket state before sending audio:

const microphone = new AgentMicrophone((data) => {
  const ws = wsRef.current;

  if (!settingsAppliedRef.current || !ws || ws.readyState !== WebSocket.OPEN) {
    return;
  }

  ws.send(data);
}, {
  sampleRate: VOICE_INPUT_SAMPLE_RATE,
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: true,
});

Without the settingsAppliedRef check, the first audio frames could be sent before Deepgram accepted the configuration, and the first turn disappeared.

RecallMEM tracks welcome, settings, settings applied, microphone start, keepalive, interruption, playback, cleanup, and reconnect events.

Each lifecycle event now maps to visible client state so I can tell where a failed session stopped.

Memory exposed the next boundary.

RecallMEM already had memory for text chat. Sending recent messages in the opening voice prompt failed when the conversation grew long, the context went stale, or the user asked for an exact detail from days earlier. RecallMEM instead exposes memory through a narrow tool.

When Deepgram sends a FunctionCallRequest, the browser calls RecallMEM’s memory endpoint. The app searches memory, formats the result, and sends a FunctionCallResponse back to Deepgram.

sendJsonMessage({
  type: "FunctionCallResponse",
  id: fn.id,
  name: fn.name,
  content,
});

The browser never touches Postgres. Server routes do. The memory tool route is the only part of the live tool-call flow that queries memory.

That route combines exact keyword search and semantic search. It also has a timeout on purpose.

I added a retrieval timeout because a long pause sounded like the agent had stopped working.

One of my worst latency bugs came from starting a voice session inside a long chat and sending too much recent transcript context into Deepgram. It was technically “more context.” It was also worse. The fix was to keep startup context small: a few compact recent messages, shorter profile/rules text, fewer memory facts, then let search_memory pull older detail only when needed.

Startup now sends a small recent context and retrieves older details only when needed.

If voice turns do not save back into the normal chat, the app has already split in two.

In RecallMEM, when Deepgram emits ConversationText, the app appends it into the same message list normal chat uses. Assistant turns go through the normal save path. Memory extraction runs after that, the same way it does for text.

Voice turns therefore enter the same fact-extraction and future-recall path as typed turns.

The first working version was not done.

The agent greeted out loud. Great. Then later turns came back as text only. Not great.

At first it looked like a model problem, or maybe a Deepgram problem, or maybe one of those haunted browser-audio bugs that make you question your career choices.

It was simpler than that. I had tied “ready for audio” too tightly to one event. The client needed to unlock playback on more than one valid signal:

case "AgentThinking":
  allowNextAgentAudio();
  setStatus("thinking");
  break;

case "AgentStartedSpeaking":
  stopPlayback(false);
  setStatus("speaking");
  break;

case "ConversationText":
  if (role === "assistant") allowNextAgentAudio();
  appendConversationText(role, content || "");
  break;

The client now unlocks playback from every valid agent-audio signal.

After the greeting worked, interruptions, stale audio, overlapping chunks, tool-call pauses, reconnects, sample-rate mismatches, and noisy rooms still broke later turns.

The mute button moved onto the required list after I tried the agent in a loud room and could not stop it from reacting.

I would use this architecture when voice needs the same tools, memory, transcripts, settings, and user context as an existing text experience.

For a one-off bot with no persistent state, the smaller record-transcribe-respond-speak loop is enough.

Voice is another interface into the same product. Building it as a separate system only postpones the moment a user asks it for something the text app already knows.

That user expects the transcript to save, memory to update, and settings to carry over.

Deepgram supplies the realtime voice loop.

RecallMEM keeps memory and tools behind server routes, issues a short-lived browser token, and saves voice turns through the normal transcript path.

Questions about this post? Ask the terminal on my homepage — it knows this whole site.

Chris Dabatos

Staff DevRel Engineer @ Fly.io, AI Engineer, and Technical Storyteller based in Las Vegas. He builds things with AI and writes about what breaks.