> ## Documentation Index
> Fetch the complete documentation index at: https://docs.irisagent.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AutoEval: generate evals from LLM traffic

> Build a golden eval set from your LLM call logs, keep it current, and use it to catch regressions and compare models.

AutoEval turns the LLM calls your apps already make into a **golden eval set** for each use case, and keeps it up to date as your traffic changes. Nobody has to write or maintain test prompts.

Use it to:

* **Compare models** (including fine-tuned or self-hosted ones) on correctness, cost, and latency before you switch.
* **Catch regressions** by scoring every production call in the [dashboard](#view-scores-in-the-dashboard).
* **Test changes before release** from your test suite or CI with the [evaluate API](#evaluate-api).

AutoEval is part of [Iris Control](https://docs.irisagent.com/iris-control.md). Evals run after each call completes, never on your request path, so they add no latency. AutoEval never changes the models your app uses.

> [!TIP]
>
> Try the AutoEval tab of the [Iris Control live demo](https://irisagent.com/iris-control-demo/#autoeval) before you connect anything.

## Get started

AutoEval needs the input and output of each LLM call. Pick one option.

### Option 1: Route calls through Iris Control (recommended)

Every call is captured automatically, and you also get Iris Control's routing, failover, spend limits, and redaction.

1. In the Iris Control dashboard, [create an API key](https://docs.irisagent.com/iris-control.md#step-1-get-an-api-key) and [add your model providers](https://docs.irisagent.com/iris-control.md#step-2-add-your-model-providers).

2. [Create a use case](https://docs.irisagent.com/iris-control.md#step-3-create-a-use-case) for each LLM feature, such as `support-answers`.

3. Point your OpenAI-compatible client at Iris Control and set `model` to the use case name:

   ```python Python theme={null}
   from openai import OpenAI

   client = OpenAI(
       base_url="https://api1.irisagent.com/v1/control",
       api_key=IRIS_API_KEY,
   )

   response = client.chat.completions.create(
       model="support-answers",
       messages=[{"role": "user", "content": "How do I export last month's invoices?"}],
   )
   ```

4. Optional: [add custom rules](#custom-rules).

Calls appear in the [call log](https://docs.irisagent.com/iris-control.md#call-log), and scores appear on the **AutoEval** page. For Node.js, cURL, and headers, see the [Iris Control quickstart](https://docs.irisagent.com/iris-control.md#quickstart).

### Option 2: Send a copy of your call logs

Your app keeps calling its providers as it does today. You send AutoEval a copy of the calls. See [Send your call logs](#send-your-call-logs).

## Golden eval set

AutoEval builds one set per use case from your traffic:

* Inputs come from real calls.
* Duplicates and near-duplicates are removed.
* The set is balanced across workflows, tenants, languages, and edge cases, and includes failed calls.
* New use cases and workflows get their own slice. Inputs that no longer match your product are retired.

You can also add specific calls from the call log, or upload a file of inputs every model must pass.

## Rules

### Default rules

Every output is graded for correctness with no setup: is it a correct, complete answer to its own system prompt and user prompt? You get a verdict (`correct` or `defect`), a score from 0 to 5, and an explanation.

### Custom rules

Custom rules are plain-English checks, in the same format as [AutoQA](https://docs.irisagent.com/configuring-ai-and-automation/AutoQA.md) rules. Each use case has its own rules.

To add one, open **AutoEval** in the Iris Control dashboard, select the use case, and click **Create rule**.

| Field | Description |
| - | - |
| **Description** | What the output must or must not do. |
| **Category** | Groups rules in reports, such as Format, Accuracy, Compliance, Tone, or Custom. |
| **Severity** | `critical`, `warning`, or `info`. Critical failures show first. |

Examples:

| Category | Rule |
| - | - |
| Format | The output must include at least one link to a help center article. |
| Format | The output must be valid JSON with the fields `intent` and `priority`. |
| Language | Reply in the same language, or mix of languages, the user wrote in. |
| Compliance | Never promise a refund amount or a delivery date. |
| Accuracy | Never mention a product feature that is not described in the system prompt. |

A rule returns `not_applicable` when it doesn't apply to a call (for example, a refund rule on a password question). These results are left out of the pass rate.

## Results

Each graded output has one result:

```json theme={null}
{
  "id": "call_8f2a",
  "model": "gpt-5.6-sol",
  "judge": "gemini-3.7-flash",
  "correctness": {
    "verdict": "defect",
    "score": 2,
    "explanation": "The user asked how to export invoices; the answer explains how to download a single receipt."
  },
  "rules": [
    {
      "ruleId": "include-link",
      "outcome": "fail",
      "explanation": "The output contains no link."
    },
    {
      "ruleId": "same-language",
      "outcome": "pass",
      "explanation": "The user wrote in English and the reply is in English."
    }
  ]
}
```

| Field | Description |
| - | - |
| `correctness.verdict` | `correct` or `defect`. |
| `correctness.score` | 0 to 5. Higher is more correct and complete. |
| `correctness.explanation` | Why the judge gave that verdict. |
| `rules[].outcome` | `pass`, `fail`, or `not_applicable`. |
| `rules[].explanation` | Why the rule passed or failed. |

## View scores in the dashboard

The **AutoEval** page shows scores as calls are graded:

* **Every call:** input, output, correctness verdict and score, and each rule result with explanations.
* **Rollups:** correctness rate, average score, and each rule's pass rate, by use case, model, tenant, and rule. Critical failures show first.
* **Failures:** filter to failed calls to see the examples behind each number.

## Compare models

1. Open **AutoEval**, select a use case, and open **Compare models**.
2. Choose the eval set: the golden eval set (default), calls from the call log, a filter (such as the last 30 days, or only failed calls), or an uploaded file.
3. Choose the models. Any model from your [providers](https://docs.irisagent.com/iris-control.md#step-2-add-your-model-providers) works, including a self-hosted model with an OpenAI-compatible endpoint.
4. Run the comparison.

Each input goes to each model with its original system prompt. The report shows, per model: correctness rate and average score, each rule's pass rate, cost per call and total cost, and typical and slowest latency. Open any row to compare outputs side by side.

## Send your call logs

Send one JSONL record per LLM call:

```json theme={null}
{
  "id": "call_8f2a",
  "timestamp": "2026-10-15T18:04:11Z",
  "use_case": "support-answers",
  "model": "gpt-5.6-sol",
  "messages": [
    {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
    {"role": "user", "content": "How do I export last month's invoices?"}
  ],
  "output": "Go to Billing, then Invoice history, and select Export.",
  "latency_ms": 1840,
  "metadata": {"tenant": "acme-corp"}
}
```

| Field | Required | Description |
| - | - | - |
| `id` | Yes | Your ID for the call. |
| `messages` | Yes | The input in OpenAI message format: system and user prompts, or the whole conversation before the output. |
| `output` | Yes | The model's output. |
| `model` | Yes | The model or model version that produced the output. |
| `use_case` | No | Selects the rules to grade against. Defaults to `default`. |
| `timestamp`, `latency_ms` | No | Used in reports and filters. |
| `metadata` | No | Tags to filter reports by, such as tenant, product, or language. |

For a conversation, send one record per assistant turn: the conversation so far in `messages` and the turn in `output`. For voice, use the transcript.

To start, send a one-time file to your Iris Control point of contact. To stay current, set up a recurring export to a storage bucket with them. Exports from Langfuse, LangSmith, or Helicone can be converted to this format.

## Evaluate API

Grade one output on demand, for example in CI before you ship a prompt change. Production scores don't need the API. Send `rules`, or send `use_case` to use that use case's rules.

```bash theme={null}
curl https://api1.irisagent.com/v1/control/autoeval/evaluate \
  -H "Authorization: Bearer $IRIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "system", "content": "You are the support assistant for Acme Billing. Always link the help article you used."},
      {"role": "user", "content": "How do I export last month'\''s invoices?"}
    ],
    "output": "Go to Billing, then Invoice history, and select Export.",
    "rules": [
      {"id": "include-link", "description": "The output must include at least one link to a help center article.", "category": "Format", "severity": "critical"}
    ]
  }'
```

The response has the `correctness` and `rules` fields from [Results](#results). Send up to 100 rules per request.

## Data handling

Inputs and outputs are redacted for PII and secrets before they are stored. Set retention with the use case's `logging.retention_days`. If prompts and outputs can't leave your environment, Iris Control and AutoEval can run in your own VPC. See the [customer VPC reference architecture](https://docs.irisagent.com/security-and-compliance/Reference-Architecture-Customer-VPC.md).

Questions? Email [contact@irisagent.com](mailto:contact@irisagent.com) or [book an Iris Control demo](https://irisagent.com/get-a-demo-iris-control/).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.