ChatGPT Memory architecture: long context, vector retrieval, and cross-session persistence

ChatGPT Memory is not magic - it is the combination of three layers: 1.05M token long context + vector retrieval (RAG) + cross-session persistence. This article breaks down all three from the architect's perspective, with selection guidance, write strategy, compression algorithms, and token cost control.

Memory's three-layer architecture

+-------------------------------------------+
|  Layer 1: Long context (1.05M token)      |
|  - Recent N turns of original dialog       |
|  - Current session summary                 |
|  - User preference cache                   |
+-------------------------------------------+
              ^ recall / write
+-------------------------------------------+
|  Layer 2: Vector retrieval layer           |
|  - Semantic match (user query -> memory)   |
|  - top-K recall + rerank                   |
|  - Return (memory_id, text, score, ts)     |
+-------------------------------------------+
              ^ persist
+-------------------------------------------+
|  Layer 3: Persistence storage             |
|  - Vector store (pgvector / Pinecone)      |
|  - KV store (preferences / settings)      |
|  - Full dialog archive (retraining / audit)|
+-------------------------------------------+

Each layer has a distinct job:

• Layer 1 (long context): the model's recall window. 1.05M looks big, but filling it on every call is expensive, so it only carries 'recent N turns + current summary'.
• Layer 2 (vector retrieval): on-demand recall of historical memory. When Layer 1 cannot hold everything, fetch top-K relevant items from the vector store and stuff into the prompt.
• Layer 3 (persistence): memory data lands on disk. New conversation arrives -> summary -> write; user deletes -> delete; cross-device sync.

Core design principle - memory is not for replacing prompt; it is for reducing repeated input and retaining cross-session context.

Vector store selection

Production choices by size:

pgvector in practice:

-- Enable pgvector
CREATE EXTENSION IF NOT EXISTS vector;

-- memories table
CREATE TABLE memories (
  id BIGSERIAL PRIMARY KEY,
  user_id TEXT NOT NULL,
  conversation_id TEXT,
  memory_text TEXT NOT NULL,
  embedding vector(3072),  -- text-embedding-3-large dims
  memory_type TEXT,        -- 'preference' / 'fact' / 'summary'
  confidence FLOAT,
  created_at TIMESTAMPTZ DEFAULT now(),
  version INTEGER DEFAULT 1
);

-- HNSW index (fast approximate NN)
CREATE INDEX ON memories USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

Recall SQL:

-- top-5 recall
SELECT id, memory_text, memory_type,
       1 - (embedding <=> $1) AS similarity
FROM memories
WHERE user_id = $2
  AND version = (SELECT MAX(version) FROM memories WHERE user_id = $2)
ORDER BY embedding <=> $1
LIMIT 5;

In production, pgvector + HNSW on 100K vectors gives under 10ms recall.

Memory write strategy

Wrong approach: write every turn. One 50-turn conversation writes 50 memory items - storage explodes, mostly noise.

Correct approach: end-of-conversation three-segment summary, write once.

def summarize_and_store(conversation):
    response = client.responses.create(
        model="gpt-5.6-luna",  # cheapest model for summary
        input=[
            {"role": "system", "content": """You are a memory summarizer. Given a conversation, output three sections:

1. User preferences: habits, preferences, style requirements expressed by the user
2. Key facts: factual information the user mentioned (address / work / friend names)
3. Dialog summary: 1-3 sentence summary of the conversation's core content

Each section is a JSON array. Each item <= 200 chars. Only keep high-confidence items; mark low-confidence ones as skipped and not output."""},
            {"role": "user", "content": format_conversation(conversation)},
        ],
    )

    result = parse_summary(response.output_text)

    # Write to vector store
    for memory in result.preferences + result.facts + [result.summary]:
        if memory.confidence < 0.7:
            continue
        embed = embed(memory.text)
        db.insert_memory(
            user_id=conversation.user_id,
            memory_text=memory.text,
            embedding=embed,
            memory_type=memory.type,
            confidence=memory.confidence,
            conversation_id=conversation.id,
        )

Key design:

• Use the cheapest model (gpt-5.6-luna) for summary - cost saving.
• confidence threshold filters noise (do not write under 0.7).
• Three-segment is more structured than a single summary; recall can filter by type for precision.

Context compression strategy

1.05M tokens cannot be stuffed into every prompt. Three compression strategies stacked:

Sliding window (recent N turns)

# Last 20 turns of dialog, original text
recent_turns = conversation.messages[-40:]  # 20 turn = 40 message

Fixed window, simple and reliable, good for real-time conversation.

Hierarchical summary (compress old turns)

# Older than 20 turns, replaced with summary
if len(conversation.messages) > 40:
    older_summary = summarize_older(conversation.messages[:-40])
    prompt_messages = [
        {"role": "system", "content": f"Conversation history summary: {older_summary}"},
        *recent_turns,
    ]

Refresh the summary each time the conversation ends.

On-demand recall (vector store retrieval)

# When user query arrives, recall relevant memories by query
relevant_memories = recall_top_k(user_query, user_id, k=5)

prompt_messages = [
    {"role": "system", "content": f"User historical relevant memories: {format_memories(relevant_memories)}"},
    {"role": "system", "content": f"Conversation history summary: {older_summary}"},
    *recent_turns,
    {"role": "user", "content": user_query},
]

Recall only top-K, not everything.

Token cost control

A Memory system's cost has three components:

# Assume each call's prompt ~ 50K tokens (20 turns + summary + 5 memory)
# GPT-5.6 Terra: $2.50/$15 per MTok
cost_per_call = (50 * 0.001 * 2.5) + (1 * 0.001 * 15)  # ~ $0.14

# 1M calls/month = $140K
monthly_cost = 1_000_000 * 0.14  # $140K

Optimizations:

All four together, monthly cost drops from $140K to ~$35K.

Cross-session consistency

Keeping memory consistent across devices / sessions requires versioning:

# Write with version
db.insert_memory(
    user_id=user_id,
    memory_text=memory.text,
    embedding=embed,
    version=current_version + 1,  # monotonically increasing
    created_at=now(),
)

# Recall only the latest version
SELECT ... WHERE user_id = $1
  AND version = (
    SELECT MAX(version) FROM memories WHERE user_id = $1
  )

Key design:

• Every memory carries (user_id, version, created_at, conversation_id).
• Recall only the latest version; old memory is soft-deleted (deleted=true flag).
• User can 'rollback' - undo the most recent memory write.

Common pitfalls

1. Writing every turn: storage explodes + noise dominates. Use end-of-conversation summary instead.
2. No confidence label: low-quality memory pollutes recall. Force the LLM to score confidence on output.
3. Stuffing uncompressed into prompt: 1.05M looks big but cost is high. Use hierarchical summary + sliding window.
4. Cross-session version conflict: user A syncs memory between phone and laptop. Solve with version + timestamp.
5. GDPR non-compliance: EU users have the right to delete all their memory data. Architecture must support user_id-level bulk delete.
6. Recalled memory treated as fact: memory is user/model-generated, may be wrong. Tell the model in the prompt that memory is 'what the user said in the past', not objective truth.

Next steps

• Want the ChatGPT UI Memory usage? Read ChatGPT Memory Complete Manual.
• Want real Memory use cases? Read ChatGPT Memory: 12 high-value use cases from tech stack to family info.
• Curious about GPT-5.6 family long-context capability? Read The complete guide to GPT models (2026-07): GPT-5.6 Sol, Terra, Luna.