Prompt evaluation and failure debugging: LLM-as-judge + regression tests + Debug mode

Writing the prompt is just the start - after it goes to production you need: LLM-as-judge auto-scoring, regression test sets, failure case taxonomy, Debug logs, cost visualization. This article gives a complete 'prompt launch + continuous monitoring' pipeline.

Why evaluate?

A Prompt in production faces three problems:

1. Stability: model version updates / prompt tweaks can cause output drift.
2. Regression: changing prompt to fix one case may break 5 others.
3. Failure debugging: user reports 'AI answered wrong' - what's specifically wrong? how to fix?

Evaluation is not 'done after writing prompt', it's a daily product practice.

1. LLM-as-judge auto-scoring

Let GPT-5.6 act as judge and score prompt outputs. Key principle: judge uses a stronger model (Sol), contestant uses whatever.

Mode A: score (0-10)

import openai

client = openai.OpenAI()

JUDGE_PROMPT = """
You are a prompt output evaluation expert. Given input, prompt output, reference answer, score 0-10:

- 0-3: completely wrong / off-topic / hallucination
- 4-6: partially correct but with notable issues
- 7-9: mostly correct, minor flaws
- 10: completely correct and matches prompt style

Output JSON: {"score": <int>, "reason": "<one sentence>"}

Input: {input}
Output: {output}
Reference: {reference}
"""

def judge(input_text, output, reference):
    response = client.responses.create(
        model="gpt-5.6-sol",  # use Sol as judge
        input=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                input=input_text,
                output=output,
                reference=reference,
            ),
        }],
        text={
            "format": {
                "type": "json_schema",
                "name": "judge_result",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "score": {"type": "integer", "minimum": 0, "maximum": 10},
                        "reason": {"type": "string"},
                    },
                    "required": ["score", "reason"],
                    "additionalProperties": False,
                },
            },
        },
    )
    return json.loads(response.output_text)

# Usage
result = judge("What is GPT-5.6?", "GPT-5.6 is OpenAI's flagship model family released 2026-07-09.", "OpenAI flagship model")
print(result["score"], result["reason"])

Mode B: pairwise (A vs B)

Useful when revising prompts - A vs B, which is better?

PAIRWISE_PROMPT = """
You are a prompt output evaluation expert. Given the same input's two outputs (A and B), judge which is better.

Evaluate: accuracy / fluency / style / format compliance.
Mark winner as 'A' / 'B' / 'tie'.
One-sentence reason.

Input: {input}
Output A: {output_a}
Output B: {output_b}
"""

def pairwise(input_text, output_a, output_b):
    response = client.responses.create(
        model="gpt-5.6-sol",
        input=[{
            "role": "user",
            "content": PAIRWISE_PROMPT.format(
                input=input_text,
                output_a=output_a,
                output_b=output_b,
            ),
        }],
        text={
            "format": {
                "type": "json_schema",
                "name": "pairwise_result",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "winner": {"type": "enum", "values": ["A", "B", "tie"]},
                        "reason": {"type": "string"},
                    },
                    "required": ["winner", "reason"],
                    "additionalProperties": False,
                },
            },
        },
    )
    return json.loads(response.output_text)

Usage: run 50+ cases, new prompt wins ≥ 60% in pairwise to consider launch.

Mode C: rubric (multi-dimension)

RUBRIC_PROMPT = """
You are a prompt output evaluation expert. Given input and output, score (0-10 each):

1. Accuracy (factual correctness): is the output factually correct?
2. Fluency: is the language natural?
3. Style match: does it match the prompt's required style?
4. Format compliance: does it match the prompt's required format?

Score each + 1 sentence reason. Overall = weighted (accuracy 0.4 + fluency 0.2 + style 0.2 + format 0.2).

Output JSON strictly by schema.

Input: {input}
Output: {output}
"""

Usage: when you need quality breakdown - e.g. customer support prompt, accuracy is most important (40% weight), others balanced.

2. Regression test sets

Rerun regression before every prompt change, avoid breaking old cases.

# tests/prompts/customer-support.jsonl
{"id": "happy-001", "input": "my order 12345 hasn't arrived", "category": "happy", "expected": "check order status then reply"}
{"id": "happy-002", "input": "how do I cancel?", "category": "happy", "expected": "explain cancel flow"}
{"id": "edge-001", "input": "I want to complain!", "category": "edge", "expected": "apologize + transfer to human"}
{"id": "edge-002", "input": "is 1111111111111111 an order number?", "category": "edge", "expected": "tell user order number format is wrong"}
{"id": "failure-001", "input": "what company is OpenAI?", "category": "failure", "expected": "correctly answer OpenAI company info"}
{"id": "failure-002", "input": "GPT-5.6 release date?", "category": "failure", "expected": "2026-07-09, no fabrication"}

Config:

• Count: 50-200 cases
• Distribution: 30% happy path + 40% edge cases + 30% historical failures
• Format: JSONL, each row has input / category / expected
• Storage: git versioned (tests/prompts/*.jsonl)

import json

def run_regression(prompt_template, test_file, judge_fn):
    with open(test_file) as f:
        cases = [json.loads(line) for line in f]

    results = []
    for case in cases:
        output = run_prompt(prompt_template, case["input"])
        score = judge(case["input"], output, case["expected"])
        results.append({
            "id": case["id"],
            "category": case["category"],
            "score": score["score"],
            "reason": score["reason"],
        })

    avg_score = sum(r["score"] for r in results) / len(results)
    failed = [r for r in results if r["score"] < 7]

    print(f"Avg score: {avg_score:.2f}")
    print(f"Failed cases: {len(failed)}/{len(results)}")
    for r in failed:
        print(f"  {r['id']}: {r['reason']}")

    return avg_score, failed

3. Failure case taxonomy

Each failure case must be categorized:

Auto classifier:

FAILURE_CATEGORIES = ["hallucination", "off_topic", "format_error", "length_overrun", "style_mismatch", "timeout"]

def classify_failure(input_text, output, reference):
    response = client.responses.create(
        model="gpt-5.6-sol",
        input=[{
            "role": "user",
            "content": f"""Determine which failure category the prompt output belongs to:

Input: {input_text}
Output: {output}
Reference: {reference}

Failure categories: {FAILURE_CATEGORIES}
""",
        }],
        text={
            "format": {
                "type": "json_schema",
                "name": "failure_classification",
                "strict": True,
                "schema": {
                    "type": "object",
                    "properties": {
                        "category": {"type": "enum", "values": FAILURE_CATEGORIES},
                        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
                    },
                    "required": ["category", "confidence"],
                    "additionalProperties": False,
                },
            },
        },
    )
    return json.loads(response.output_text)

# Usage: auto-archive failure cases
result = classify_failure(input, output, reference)
# {"category": "hallucination", "confidence": 0.92}

4. Debug mode

Each call must log key fields:

import hashlib
import json
import time

def run_prompt_with_debug(prompt_template, input_data, **kwargs):
    # Compute input hash (dedup + regression set correlation)
    input_hash = hashlib.md5(json.dumps(input_data, sort_keys=True).encode()).hexdigest()

    # Render prompt
    full_prompt = prompt_template.format(**input_data)

    # Call + time
    start = time.time()
    response = client.responses.create(
        model="gpt-5.6-terra",
        input=[{"role": "user", "content": full_prompt}],
        **kwargs,
    )
    latency_ms = (time.time() - start) * 1000

    # Log full data
    log_entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "input_hash": input_hash,
        "input": input_data,
        "full_prompt": full_prompt,    # full prompt, don't log only diff
        "output": response.output_text,
        "token_usage": {
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
            "cached_tokens": response.usage.input_tokens_details.cached_tokens,
        },
        "latency_ms": latency_ms,
        "model": response.model,
        "model_version": "gpt-5.6-terra-2026-08-08",  # lock specific version
    }

    # Auto-run judge + archive failure cases
    if "reference" in input_data:
        judge_result = judge(input_data, response.output_text, input_data["reference"])
        log_entry["judge_score"] = judge_result["score"]

        if judge_result["score"] < 7:
            # auto-archive failure case
            archive_failure(log_entry)

    save_log(log_entry)
    return response.output_text

Key fields:

• input_hash: dedup + regression set correlation
• full_prompt: full prompt (don't log only diff, debugging needs it)
• token_usage: input + cached + output separately
• latency_ms: from call start to response
• model_version: lock specific version (not alias)
• judge_score: auto score

5. Cost visualization

Compute monthly cost per prompt:

# Assume
prompt_id = "customer-support-v3"
monthly_calls = 100_000
avg_input_tokens = 800
avg_output_tokens = 200
cache_hit_rate = 0.4  # 40% calls hit cache
model = "gpt-5.6-terra"

# GPT-5.6 Terra: $2.50/$15 per MTok, cache 1/4
non_cached_input_cost = avg_input_tokens * (1 - cache_hit_rate) * 2.50 / 1_000_000
cached_input_cost = avg_input_tokens * cache_hit_rate * (2.50 / 4) / 1_000_000
output_cost = avg_output_tokens * 15 / 1_000_000

cost_per_call = non_cached_input_cost + cached_input_cost + output_cost
monthly_cost = cost_per_call * monthly_calls

print(f"Cost per call: ${cost_per_call:.4f}")
print(f"Monthly cost: ${monthly_cost:.2f}")

# Monitoring metrics
# - monthly cost trend (rise > 30% alert)
# - single-call cost > threshold alert (possible abuse)
# - cache hit rate drop alert (prompt may have changed)

6. A/B launch + gray rollout

New prompt launch flow:

Local regression set OK  →  A/B 10% traffic 1 week  →  gray 50% traffic 1 week  →  full launch
                          ↑ judge score +0.5+       ↑ business metrics aligned    ↑

def ab_test_prompts(input_data, old_prompt, new_prompt, traffic_split=0.5):
    """A/B test with traffic split"""
    bucket = hash(input_data["user_id"]) % 100  # 0-99
    if bucket < traffic_split * 100:
        # A bucket
        return run_prompt(old_prompt, input_data, log_extra={"variant": "A"})
    else:
        # B bucket
        return run_prompt(new_prompt, input_data, log_extra={"variant": "B"})

A/B must-watch metrics:

• judge score (must new prompt ≥ old +0.5)
• business metrics (user satisfaction, task completion, bounce rate)
• cost (must not rise > 50%)

A/B 1-2 weeks: new prompt must not underperform old on all metrics to full launch.

7. Complete pipeline diagram

[Prompt revision]
     ↓
[Local regression set run] ← tests/prompts/*.jsonl
     ↓
[judge score all ≥ 7]? - no → revise prompt retry
     ↓ yes
[A/B 10% traffic]
     ↓
[1 week later metrics] - judge / business / cost aligned? - no → rollback
     ↓ yes
[Gray 50% traffic]
     ↓
[1 week later full launch]

FAQ

1. Is LLM-as-judge or human evaluation more accurate?

Both. (1) LLM-as-judge has high volume, low cost, automated - fits daily 1000+; (2) human evaluation more accurate but expensive - fits monthly 50-200 key cases for calibration. Best practice: LLM-as-judge for daily gate, human every 2 weeks for calibration - target correlation > 0.8.

2. Where to put regression test sets?

Three places: (1) code repo at tests/prompts/<feature>.jsonl - git versioned + CI auto-run; (2) Prompt management platform; (3) database table. Production recommends (1) + (2).

3. How to judge prompt revision is good?

Three steps: (1) regression set compare judge score (must +0.5+ and not break old); (2) A/B 10% traffic; (3) gray 1-2 weeks full launch. Note: A/B must serve same user group.

4. How to debug hallucination cases?

Three steps: (1) find the specific hallucinated fact; (2) check prompt has grounding context; (3) add RAG. Common anti-pattern: model answers open facts from memory - must use RAG.

5. Evaluation cost?

Three layers: (1) evaluation cost itself; (2) evaluated prompt runtime cost; (3) evaluation frequency cost. 100 cases × 500 input token × $2.50/MTok = $0.125/day = $3.75/month. Sol judge 2-3x more expensive.

Next steps

• Want prompt core patterns? Read Prompt Engineering Core Patterns: 8 Templates That 2× GPT Output.
• Want prompt advanced? Read Prompt Engineering Advanced: Multi-Turn Context, Structured Outputs, and GPT-5.6 Tuning.
• Want coding prompts? Read GPT-5.6 coding prompt patterns: 12 templates for Codex / Cursor that get it right on the first try.