Realtime Voice Agents in production: phone support and voice assistants with Realtime API + function calling

A Realtime Voice Agent is a different beast from Chat Completions: it streams audio, starts generating while the user is still pausing, and triggers functions mid-conversation. This guide is for builders shipping phone / voice-assistant products: WebRTC vs WebSocket, VAD tuning, mid-conversation function calling, barge-in, monitoring. Production template at the end.

When to use a Realtime Voice Agent

Not every voice scenario needs Realtime - decide first.

• Phone support: user calls and asks "where is my package". AI listens and queries the order live. Use Realtime - users expect human-speed; traditional ASR+LLM+TTS pipeline with 3-5s latency feels robotic.
• Voice assistants: smart speaker / car / smart home. User says "turn the living room lights up". AI executes and replies. Use Realtime - latency matters, frequent mid-conversation function calls.
• Meeting transcription: convert recordings to text + summary. Don't use Realtime - Whisper + GPT-4o batch is fine; Realtime is overpriced here.
• Audiobooks / voiceover: pure TTS. Use TTS API; Realtime works but is more expensive.

WebRTC vs WebSocket selection

• Browser / App: WebRTC. MediaStream → RTCPeerConnection → DataChannel → Realtime API.
• Phone integration (Twilio / Vonage): WebSocket. Twilio Media Streams forwards μ-law 8kHz over wss://, you relay to Realtime API.
• IVR / self-hosted: WebSocket + SIP gateway.

server_vad tuning

Realtime API's VAD is server-side - the model decides when the user has finished. Three knobs:

session.update({
  turn_detection: {
    type: 'server_vad',
    threshold: 0.5,           // VAD trigger (0-1)
    silence_duration_ms: 350, // user pause considered 'done' (ms)
    prefix_padding_ms: 250,  // audio prefix buffer (ms)
  },
});

Per scenario:

After tuning, transcribe recordings with Whisper and measure WER - target under 5%. WER over 10% means the agent is unusable.

Mid-conversation function calling

This is the single biggest advantage of Realtime Voice Agent over traditional pipelines - the model fires functions mid-stream while TTS says "let me check".

session.update({
  tools: [
    {
      type: 'function',
      name: 'query_order',
      description: 'Query order status. Trigger: when user asks "where is my order X" or similar.',
      parameters: {
        type: 'object',
        properties: {
          order_id: { type: 'string', description: 'Order ID' },
        },
        required: ['order_id'],
      },
    },
  ],
  tool_choice: 'auto',
});

// Listen for response.function_call_arguments.done
ws.on('response.function_call_arguments.done', async (event) => {
  const args = JSON.parse(event.arguments);
  const result = await queryOrder(args.order_id);

  // Send result back to the model
  ws.send(JSON.stringify({
    type: 'conversation.item.create',
    item: {
      type: 'function_call_output',
      call_id: event.call_id,
      output: JSON.stringify(result),
    },
  }));
});

End-to-end:

User: Can you check order 12345?
AI:   Sure, let me check.        ← TTS starts immediately (does not wait for function)
      [function_call: query_order(12345)]  ← model fires tool simultaneously
      [function returns: arrived at XX hub, expected this afternoon]
AI:   Your order is at XX hub, arriving this afternoon.

Hallucination guard: three layers -

1. Function descriptions spell out trigger conditions to prevent misuse.
2. Treat function returns as untrusted input. Validate order IDs, amounts, addresses.
3. High-risk actions (refunds, address changes) require explicit user confirmation. AI says "I will change your address to XX" and waits for "confirm" before executing.

Barge-in

User interruptions are part of natural conversation - you say "your order is at XX hub, expected..." and user jumps in "no wait, I need to change the address". Handle it:

session.update({
  turn_detection: {
    type: 'server_vad',
    interrupt_response: true,  // enable barge-in
  },
});

Barge-in flow:

1. User starts talking → server_vad detects
2. Stop TTS output immediately
3. Reset audio stream to last user turn (drop AI's unfinished spoken output)
4. Continue processing user's new input

Barge-in delay under 200ms feels natural. Over 500ms and the user feels they "cannot get a word in".

Call quality monitoring

Production-grade Realtime Voice Agent must monitor three numbers:

# pseudocode
def record_call(call_id):
    return {
        'ttfb_ms': time_to_first_byte,         # user-stopped to AI-started
        'turn_latency_ms': turn_latency,       # whole-turn latency
        'wer': compute_wer(audio, transcript), # ASR accuracy
        'interrupt_count': interrupt_count,    # barge-ins (>5/min means AI too verbose)
    }

WER > 5% means VAD or microphone issues; turn latency > 2s means Realtime API rate limiting or network issues.

Production template: Twilio + Realtime

import { WebSocketServer } from 'ws';
import OpenAI from 'openai';

const wss = new WebSocketServer({ port: 8080, path: '/realtime-twilio' });

wss.on('connection', async (twilioWs) => {
  // 1. Connect to OpenAI Realtime
  const openaiWs = new WebSocket(
    'wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1',
    { headers: { 'Authorization': `Bearer ${process.env.OPENAI_API_KEY}` } }
  );

  // 2. session config
  openaiWs.send(JSON.stringify({
    type: 'session.update',
    session: {
      voice: 'alloy',
      turn_detection: {
        type: 'server_vad',
        threshold: 0.5,
        silence_duration_ms: 350,
        prefix_padding_ms: 250,
        interrupt_response: true,
      },
      tools: [{
        type: 'function',
        name: 'query_order',
        description: 'Query order status',
        parameters: { /* ... */ },
      }],
      input_audio_format: 'g711_ulaw',   // Twilio is μ-law 8kHz
      output_audio_format: 'pcm16',
    },
  }));

  // 3. Twilio → OpenAI
  twilioWs.on('message', (data) => {
    const msg = JSON.parse(data);
    if (msg.event === 'media') {
      openaiWs.send(JSON.stringify({
        type: 'input_audio_buffer.append',
        audio: msg.media.payload,  // base64 μ-law
      }));
    }
  });

  // 4. OpenAI → Twilio
  openaiWs.on('message', (data) => {
    const msg = JSON.parse(data);
    if (msg.type === 'response.audio.delta') {
      twilioWs.send(JSON.stringify({
        event: 'media',
        streamSid: '...',
        media: { payload: msg.delta },
      }));
    }
  });
});

Common pitfalls

1. Codec mismatch: Twilio is g711_ulaw 8kHz, OpenAI Realtime defaults to pcm16 24kHz. Without resampling it sounds robotic. Fix: set input_audio_format: 'g711_ulaw' in session.update - OpenAI handles resampling.
2. AI repeats itself after barge-in: audio stream not reset after interrupt causes AI to continue its unfinished sentence. Fix: actively send response.cancel on interrupt.
3. Function call hangs: function not returning within 5s makes the model wait, user thinks AI is stuck. Fix: 3s function timeout + fallback ('system busy, will reply shortly').
4. WebRTC blocked by firewall: enterprise networks often block UDP. Fix: configure TURN server, fallback to WebSocket.

Next steps

• New to Realtime API? Read Realtime Voice Complete Guide: gpt-realtime and Voice Mode.
• Latency obsessed? Read Advanced Voice Deep Manual: GPT-Realtime-2.1 in production and latency tuning.
• Curious about the model family? Read The complete guide to GPT models (2026-07): GPT-5.6 Sol, Terra, Luna.