Realtime Voice guide: gpt-realtime and Voice Mode

GPT-Realtime-2.1, released by OpenAI on 2026-07-06, is a low-latency voice model that ingests an audio stream and generates spoken replies with end-to-end latency around 300ms; GPT-Realtime-2.1-mini is the low-cost variant. Both are exposed to developers through the Realtime API and power ChatGPT Voice Mode under the hood.

1. Overview

Realtime API and ChatGPT Voice Mode share the same GPT-Realtime-2.1 model family but target different consumers: Voice Mode is a product feature built into ChatGPT; the Realtime API is a developer-facing interface that exposes the model over WebRTC or WebSocket so you can build your own voice product. This guide covers the difference between the two, WebRTC vs WebSocket selection, ephemeral token auth, and mid-conversation function calling end to end. All code is based on the Realtime API as of 2026-07, cross-referenced with the official OpenAI WebRTC guide.

2. Key points

• Model family: GPT-Realtime-2.1 (flagship) and GPT-Realtime-2.1-mini (low cost), released 2026-07-06. The legacy Realtime API Beta was retired on 2026-05-12.
• Transport: WebRTC for the browser (peer-to-peer, low latency, native audio pipeline); WebSocket for server-mediated scenarios (easier logging, transcoding, orchestration).
• Auth: Production must use ephemeral tokens (short-lived client secrets). Never ship a long-lived API key to the browser.
• Function calling: Mid-conversation function calling lets the model invoke tools (weather, calendar, orders) during a live call without breaking the conversation flow.
• Voices: Ten built-in voices — alloy, ash, ballad, coral, echo, sage, shimmer, verse, marin, cedar; tune temperature to control prosody variation.

3. How it works

The Realtime API is a bidirectional streaming protocol. The core flow has three steps:

1. Create a session: Your server uses the long-lived API key to call POST /v1/realtime/sessions, receiving an ephemeral client secret (valid ~1 minute) and session configuration (model, voice, instructions, tools, etc.).
2. Establish a connection: The browser uses the ephemeral secret to connect via WebRTC (SDP offer/answer exchange) or WebSocket; once connected, audio flows bidirectionally.
3. Bidirectional audio streaming: The user's microphone audio streams to the model, and the model streams spoken replies back; function call events fire asynchronously mid-conversation.

The key difference from the traditional Chat Completions API: Realtime API is not a request-response model — it is a continuous bidirectional stream. You send audio, text, and function call results at any point during the conversation, and the model returns audio, text, and function call requests at any point.

4. Practical steps

4.1 Server: mint an ephemeral token

Never embed a long-lived API key in client code. Your server trades the real key for a short-lived token the browser can use:

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY automatically

# Create an ephemeral session and get the short-lived client secret
session = client.beta.realtime.sessions.create(
    model="gpt-realtime-2.1",
    voice="alloy",
    instructions="You are GPTMap's voice assistant. Answer concisely in English.",
)

# Only pass client_secret.value to the browser — it expires in minutes
ephemeral_token = session.client_secret.value
print(ephemeral_token)
# Return this token to the frontend via your API endpoint

The client_secret expires in 600 seconds (10 minutes) by default, with an allowed range of 10 to 7200 seconds. The browser should use it immediately to connect — do not store it.

4.2 Browser: WebRTC connection

The browser uses the ephemeral token to establish a WebRTC peer connection. Here is the minimal working example:

// 1. Fetch the ephemeral token from your server
const response = await fetch("/api/realtime-token");
const { token } = await response.json();

// 2. Create a peer connection
const pc = new RTCPeerConnection();

// Add the local microphone stream to the connection
const localStream = await navigator.mediaDevices.getUserMedia({ audio: true });
localStream.getTracks().forEach((track) => pc.addTrack(track));

// 3. Do the SDP offer/answer exchange with the ephemeral token
const dc = pc.createDataChannel("oai-events");

const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// The SDP endpoint accepts raw SDP text, not JSON
const sdpResponse = await fetch(
  "https://api.openai.com/v1/realtime/calls",
  {
    method: "POST",
    body: offer.sdp,  // raw SDP text
    headers: {
      Authorization: `Bearer ${token}`,
      "Content-Type": "application/sdp",
    },
  }
);
const answerSdp = await sdpResponse.text();
await pc.setRemoteDescription({ type: "answer", sdp: answerSdp });

// 4. Play the model's returned audio
pc.ontrack = (e) => {
  const audio = new Audio();
  audio.srcObject = e.streams[0];
  audio.play();
};

// 5. Send and receive events via the data channel
dc.onmessage = (e) => {
  const event = JSON.parse(e.data);
  console.log("Realtime event:", event.type);
};

4.3 Mid-conversation function calling

Declare tools when creating the session. When the model needs a tool during the conversation, it automatically triggers a function call:

session = client.beta.realtime.sessions.create(
    model="gpt-realtime-2.1",
    voice="alloy",
    tools=[
        {
            "type": "function",
            "name": "get_weather",
            "description": "Get the current weather for a given city",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "City name"},
                },
                "required": ["city"],
            },
        }
    ],
)

Once connected, when the user says "what's the weather in London," the model emits a function_call event via the data channel. Your code executes get_weather("London") and returns the result via conversation.item.create; the model incorporates the result into its spoken reply — the user hears only the answer, not the function call.

4.4 Choosing a voice and tuning temperature

session = client.beta.realtime.sessions.create(
    model="gpt-realtime-2.1",
    voice="nova",        # warm, leaning feminine
    temperature=0.8,     # 0.6 conservative / 0.8 balanced / 1.0 more expressive
    input_audio_transcription={
        "model": "gpt-realtime-whisper"  # server-side transcription for audit
    },
)

The ten voices group by style: grounded (alloy, echo), bright (coral, marin, sage, verse), and narrative (ash, ballad, cedar, shimmer).

5. Common errors and fixes

6. Next steps

Once you are comfortable with the Realtime API, continue with:

• ChatGPT Complete Guide (2026): From Beginner to Expert — Voice Mode on the consumer side
• OpenAI API Beginner: Your First GPT-5.6 Call — Responses API text streaming fundamentals
• The complete guide to GPT models (2026-07): GPT-5.6 Sol / Terra / Luna — model selection overview