GPT-Talk
ChatGPTGPT ModelsCodexAPIPromptMemoryMCPRelease Notes

Realtime Voice guide: gpt-realtime and Voice Mode

How to ship low-latency voice AI with OpenAI's gpt-realtime / gpt-realtime-mini: WebRTC vs WebSocket, mid-conversation function calling, and Voice Mode best practices.

TL;DR
gpt-realtime and gpt-realtime-mini are OpenAI's 2026 low-latency voice API family, supporting interruption, tone detection, and laughter. They power ChatGPT Voice Mode. This guide covers the Realtime API vs Voice Mode, choosing WebRTC vs WebSocket, and mid-conversation function calling.
gpt-realtime is OpenAI's 2026 low-latency multimodal voice model. The model ingests an audio stream and generates spoken replies with end-to-end latency around 300ms; gpt-realtime-mini is the low-cost variant.

How to

  1. Pick a transport: WebRTC for browsers, WebSocket for servers

    Browser-to-browser voice: pick WebRTC (peer-to-peer, low latency). Server-mediated or when you need logging/orchestration: pick WebSocket — server can run VAD, record, or hand off to a human.

  2. Server issues an ephemeral token to the client

    Never ship a long-lived API key to the browser. Use /v1/realtime/client_secrets on the server to mint a 1-minute token; the client connects with that token.

  3. Open the first connection and send a greeting

    Configure session.update with modalities=["audio","text"], voice="alloy", instructions="You are a friendly assistant". Then conversation.item.create + response.create to make the model speak first.

  4. Enable mid-conversation function calling

    Declare your functions in session.update's tools[]. When the model hears a fitting utterance it triggers the call; return the tool result and the model continues the conversation.

  5. Pre-launch latency and resource checks

    Use VAD to drop silence frames. Enable input_audio_transcription server-side for audit. Watch first-audio-byte latency with a <600ms target.

Realtime Voice guide: gpt-realtime and Voice Mode

gpt-realtime is OpenAI's 2026 low-latency multimodal voice model. The model ingests an audio stream and generates spoken replies with end-to-end latency around 300ms; gpt-realtime-mini is the low-cost variant.

1. Overview

gpt-realtime and gpt-realtime-mini are OpenAI's current low-latency voice API family, supporting interruption, tone detection, and even laughter. They power ChatGPT Voice Mode. This guide covers the Realtime API vs Voice Mode, choosing WebRTC vs WebSocket, and mid-conversation function calling.

2. Key points

  • Realtime API vs Voice Mode: the API is for developers and exposes gpt-realtime / gpt-realtime-mini; Voice Mode is the consumer product inside ChatGPT
  • WebRTC works in the browser with peer-to-peer low latency; WebSocket goes via your server, easier to log and orchestrate
  • Mid-conversation function calling lets the model invoke tools (weather, calendar, orders) during a live call
  • Built-in voices include alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse; tune temperature to control prosody variation
  • Production deployments must use ephemeral tokens — never ship a long-lived API key to the client

3. How it works

The sections below unpack each point. Skim alongside the OpenAI Realtime API documentation for full context.

4. Practical steps

  1. Define 'done' up front — what does success look like?
  2. Pick the right model using the comparison above.
  3. Run the minimal example end to end; record parameters and the model version.
  4. Integrate into your existing code.
  5. Monitor logs and failure modes; review weekly.

5. Common errors and fixes

  • 401: invalid or expired API key — rotate it.
  • 429: rate limit hit — enable exponential backoff retries.
  • 400: bad request — verify the model name and messages shape.
  • Poor quality output: try a different model, add few-shot examples, trim the prompt.

6. Next steps

Once you are comfortable here, continue with:

  • ChatGPT Complete Guide (2026): From Beginner to Expert
  • Prompt Engineering Core Patterns: 8 Templates That 2x GPT Output
  • OpenAI API Beginner: Your First GPT-5.6 Call

Key points

  • Realtime API vs Voice Mode: the API is for developers and exposes gpt-realtime / gpt-realtime-mini; Voice Mode is the consumer product inside ChatGPT
  • WebRTC works in the browser with peer-to-peer low latency; WebSocket goes via your server, easier to log and orchestrate
  • Mid-conversation function calling lets the model invoke tools (weather, calendar, orders) during a live call
  • Built-in voices include alloy, ash, ballad, coral, echo, fable, nova, onyx, sage, shimmer, verse; tune temperature to control prosody variation
  • Production deployments must use ephemeral tokens — never ship a long-lived API key to the client

❓ Frequently asked questions

ChatGPT Voice Mode is the consumer feature inside the ChatGPT app — open the mobile app, tap the voice icon, talk. The Realtime API is the developer-facing endpoint that exposes gpt-realtime / gpt-realtime-mini over WebRTC or WebSocket so you can build your own voice product. Voice Mode is built on top of the same model family, but you only use the API directly when you are shipping your own app.

Official references

Subscribe to GPTMap Weekly

One email every Monday: curated OpenAI updates, deep dives, and best practices. No ads, unsubscribe anytime.

GPTMap EditorialPublished 2026-07-12Updated 2026-07-14 2 min read
Test environment (EEAT)
Last tested: 2026-07-14
Model used: gpt-realtime
Voice#voice#OpenAI#gpt-realtime#guide