AutoEval: generate evals from LLM traffic

Build a golden eval set from your LLM call logs, keep it current, and use it to catch regressions and compare models.

AutoEval turns the LLM calls your apps already make into a golden eval set for each use case, and keeps it up to date as your traffic changes. Nobody has to write or maintain test prompts.

Use it to:

• Compare models (including fine-tuned or self-hosted ones) on correctness, cost, and latency before you switch.
• Catch regressions by scoring every production call in the dashboard.
• Test changes before release from your test suite or CI with the evaluate API.

AutoEval is part of Iris Control. Evals run after each call completes, never on your request path, so they add no latency. AutoEval never changes the models your app uses.

Try the AutoEval tab of the Iris Control live demo before you connect anything.

​

Get started

AutoEval needs the input and output of each LLM call. Pick one option.

​

Option 1: Route calls through Iris Control (recommended)

Every call is captured automatically, and you also get Iris Control’s routing, failover, spend limits, and redaction.

1. In the Iris Control dashboard, create an API key and add your model providers.

2. Create a use case for each LLM feature, such as support-answers.

3. Point your OpenAI-compatible client at Iris Control and set model to the use case name:

   Python

   from openai import OpenAI
   
   client = OpenAI(
       base_url="https://api1.irisagent.com/v1/control",
       api_key=IRIS_API_KEY,
   )
   
   response = client.chat.completions.create(
       model="support-answers",
       messages=[{"role": "user", "content": "How do I export last month's invoices?"}],
   )
   

4. Optional: add custom rules.

Calls appear in the call log, and scores appear on the AutoEval page. For Node.js, cURL, and headers, see the Iris Control quickstart.

​

Option 2: Send a copy of your call logs

Your app keeps calling its providers as it does today. You send AutoEval a copy of the calls. See Send your call logs.

​

Golden eval set

AutoEval builds one set per use case from your traffic:

• Inputs come from real calls.
• Duplicates and near-duplicates are removed.
• The set is balanced across workflows, tenants, languages, and edge cases, and includes failed calls.
• New use cases and workflows get their own slice. Inputs that no longer match your product are retired.

You can also add specific calls from the call log, or upload a file of inputs every model must pass.

​

Rules

​

Default rules

Every output is graded for correctness with no setup: is it a correct, complete answer to its own system prompt and user prompt? You get a verdict (correct or defect), a score from 0 to 5, and an explanation.

​

Custom rules

Custom rules are plain-English checks, in the same format as AutoQA rules. Each use case has its own rules.

To add one, open AutoEval in the Iris Control dashboard, select the use case, and click Create rule.

Field	Description
Description	What the output must or must not do.
Category	Groups rules in reports, such as Format, Accuracy, Compliance, Tone, or Custom.
Severity	critical, warning, or info. Critical failures show first.

Examples:

Category	Rule
Format	The output must include at least one link to a help center article.
Format	The output must be valid JSON with the fields intent and priority.
Language	Reply in the same language, or mix of languages, the user wrote in.
Compliance	Never promise a refund amount or a delivery date.
Accuracy	Never mention a product feature that is not described in the system prompt.

A rule returns not_applicable when it doesn’t apply to a call (for example, a refund rule on a password question). These results are left out of the pass rate.

​

Results

Each graded output has one result:

{
  "id": "call_8f2a",
  "model": "gpt-5.6-sol",
  "judge": "gemini-3.7-flash",
  "correctness": {
    "verdict": "defect",
    "score": 2,
    "explanation": "The user asked how to export invoices; the answer explains how to download a single receipt."
  },
  "rules": [
    {
      "ruleId": "include-link",
      "outcome": "fail",
      "explanation": "The output contains no link."
    },
    {
      "ruleId": "same-language",
      "outcome": "pass",
      "explanation": "The user wrote in English and the reply is in English."
    }
  ]
}

Field	Description
correctness.verdict	correct or defect.
correctness.score	0 to 5. Higher is more correct and complete.
correctness.explanation	Why the judge gave that verdict.
rules[].outcome	pass, fail, or not_applicable.
rules[].explanation	Why the rule passed or failed.

​

View scores in the dashboard

The AutoEval page shows scores as calls are graded:

• Every call: input, output, correctness verdict and score, and each rule result with explanations.
• Rollups: correctness rate, average score, and each rule’s pass rate, by use case, model, tenant, and rule. Critical failures show first.
• Failures: filter to failed calls to see the examples behind each number.

​

Compare models

1. Open AutoEval, select a use case, and open Compare models.
2. Choose the eval set: the golden eval set (default), calls from the call log, a filter (such as the last 30 days, or only failed calls), or an uploaded file.
3. Choose the models. Any model from your providers works, including a self-hosted model with an OpenAI-compatible endpoint.
4. Run the comparison.

Each input goes to each model with its original system prompt. The report shows, per model: correctness rate and average score, each rule’s pass rate, cost per call and total cost, and typical and slowest latency. Open any row to compare outputs side by side.

​

Send your call logs

Send one JSONL record per LLM call:

{
  "id": "call_8f2a",
  "timestamp": "2026-10-15T18:04:11Z",
  "use_case": "support-answers",
  "model": "gpt-5.6-sol",
  "messages": [
    {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
    {"role": "user", "content": "How do I export last month's invoices?"}
  ],
  "output": "Go to Billing, then Invoice history, and select Export.",
  "latency_ms": 1840,
  "metadata": {"tenant": "acme-corp"}
}

Field	Required	Description
id	Yes	Your ID for the call.
messages	Yes	The input in OpenAI message format: system and user prompts, or the whole conversation before the output.
output	Yes	The model’s output.
model	Yes	The model or model version that produced the output.
use_case	No	Selects the rules to grade against. Defaults to default.
timestamp, latency_ms	No	Used in reports and filters.
metadata	No	Tags to filter reports by, such as tenant, product, or language.

For a conversation, send one record per assistant turn: the conversation so far in messages and the turn in output. For voice, use the transcript.

To start, send a one-time file to your Iris Control point of contact. To stay current, set up a recurring export to a storage bucket with them. Exports from Langfuse, LangSmith, or Helicone can be converted to this format.

​

Evaluate API

Grade one output on demand, for example in CI before you ship a prompt change. Production scores don’t need the API. Send rules, or send use_case to use that use case’s rules.

curl https://api1.irisagent.com/v1/control/autoeval/evaluate \
  -H "Authorization: Bearer $IRIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
      {"role": "user", "content": "How do I export last month'\''s invoices?"}
    ],
    "output": "Go to Billing, then Invoice history, and select Export.",
    "rules": [
      {"id": "include-link", "description": "The output must include at least one link to a help center article.", "category": "Format", "severity": "critical"}
    ]
  }'

The response has the correctness and rules fields from Results. Send up to 100 rules per request.

​

Data handling

Inputs and outputs are redacted for PII and secrets before they are stored. Set retention with the use case’s logging.retention_days. If prompts and outputs can’t leave your environment, Iris Control and AutoEval can run in your own VPC. See the customer VPC reference architecture.

Questions? Email [e-mail] or book an Iris Control demo.