# Start an eval experiment

Source: <https://everruns.com/docs/api/operations/create_eval_experiment/>

[Docs](https://everruns.com/docs/)/[API reference](https://everruns.com/docs/api/)/[Evaluation & reporting](https://everruns.com/docs/api/groups/evaluation-reporting/)/Start an eval experiment

# Start an eval experiment

Preview

Start an experiment: run an eval’s cases against one to four variants and compare them case by case.

Public APIRequires the evals feature flag

POST`/v1/evals/{eval_id}/experiments`

See [authentication](https://everruns.com/docs/sre/runbooks/authentication/) for credentials and permissions.

## Parameters

### Path Parameters

**eval_id**

required

string

Prefixed public identifier

## Request Bodyrequired

Media typeapplication/json

Request to start an experiment.

object

**max_cost_usd**

Spending cap in USD for the whole experiment, split evenly between the variants’ runs.

number | null format: double

### Example

```json
10
```

**name**

Display name. Defaults to the variant labels joined with “vs”.

string | null

### Example

```plaintext
Luna vs Sol
```

**trials**

Trials per case and variant, 1 to 20 (default 3).

integer | null format: int32

\>= 1 <= 20

### Example

```json
3
```

**variants**

required

One to four variants. The first is the baseline the others are compared against.

Array\<object>

One configuration an experiment runs the dataset against.

object

**label**

required

Short name shown in comparisons, unique within the experiment.

string

### Example

```plaintext
candidate
```

**model_override**

Model for this variant, beating the target’s model.

string | null

### Example

```plaintext
gpt-6.2-sol
```

**target**

One of:

* object

  One of:

  * object

    Session creation parameters (mirrors CreateSessionRequest).

    object

    **agent_id**

    Agent to work in this session.

    string | null

    **harness_id**

    Harness for the session. If omitted, org default harness is used.

    string | null

    **harness_name**

    Addressable harness name (alternative to harness_id).

    string | null

    **max_iterations**

    Max LLM iterations per turn.

    integer | null

    **model_id**

    LLM model override.

    string | null

    **system_prompt**

    System prompt override (prepended to agent prompt).

    string | null

    **type**

    required

    string

    Allowed values: session

  * object

    Reference to a deployed app.

    object

    **app_id**

    required

    string

    **type**

    required

    string

    Allowed values: app

  * object

    Label-only target for externally-executed runs (e.g. imported from Mira).

    Carries provider/model labels and opaque params instead of session setup: external runs are ingested already-complete, so everruns never builds a session from this. Mirrors a provider-agnostic `(provider, model)` pair.

    object

    **model**

    required

    string

    **params**

    **provider**

    required

    string

    **type**

    required

    string

    Allowed values: external

* null

  Session creation parameters (mirrors CreateSessionRequest).

  object

  **agent_id**

  Agent to work in this session.

  string | null

  **harness_id**

  Harness for the session. If omitted, org default harness is used.

  string | null

  **harness_name**

  Addressable harness name (alternative to harness_id).

  string | null

  **max_iterations**

  Max LLM iterations per turn.

  integer | null

  **model_id**

  LLM model override.

  string | null

  **system_prompt**

  System prompt override (prepended to agent prompt).

  string | null

  **type**

  required

  string

  Allowed values: session

* object

  Reference to a deployed app.

  object

  **app_id**

  required

  string

  **type**

  required

  string

  Allowed values: app

* object

  Label-only target for externally-executed runs (e.g. imported from Mira).

  Carries provider/model labels and opaque params instead of session setup: external runs are ingested already-complete, so everruns never builds a session from this. Mirrors a provider-agnostic `(provider, model)` pair.

  object

  **model**

  required

  string

  **params**

  **provider**

  required

  string

  **type**

  required

  string

  Allowed values: external

* object

  null

## Responses

201Created

### 201

Created

Media typeapplication/json

An experiment: one dataset version against several variants.

object

**baseline**

required

Label of the variant the others are compared against.

string

**comparison**

One of:

* object

  Paired comparison by case (detail view).

  object

  **baseline**

  required

  string

  **cases**

  required

  Cases whose result changed come first.

  Array\<object>

  One case across the variants.

  object

  **case_id**

  required

  string

  **case_name**

  required

  string

  **cells**

  required

  One cell per variant, in variant order.

  Array\<object>

  One case under one variant.

  object

  **cost_usd**

  required

  What the case cost, all its trials together.

  number format: double

  **label**

  required

  string

  **passed**

  required

  Trials that passed, of `trials` that finished.

  integer format: int32

  **trials**

  required

  integer format: int32

  **changed**

  required

  Some variant’s pass rate on this case differs from the baseline’s.

  boolean

  **verdicts**

  required

  One verdict per variant other than the baseline.

  Array\<object>

  A variant against the baseline.

  object

  **broke**

  required

  Cases the baseline passes every trial of and this variant does not.

  integer format: int32

  **fixed**

  required

  Cases this variant passes every trial of and the baseline did not.

  integer format: int32

  **interval_high_points**

  number | null format: double

  **interval_low_points**

  95% interval of that difference (paired bootstrap), in points.

  number | null format: double

  **label**

  required

  string

  **paired_cases**

  required

  Cases both ran.

  integer format: int32

  **pass_rate_delta_points**

  Mean per-case pass rate difference, in percentage points.

  number | null format: double

  **verdict**

  required

  How a variant compares with the baseline.

  string

  Allowed values: improvement regression within_noise not_enough_data

* null

  null

**created_at**

required

string format: date-time

**dataset_version**

required

The eval’s `version` the experiment ran.

integer format: int32

**eval_id**

required

The eval whose cases are the dataset.

string

**id**

required

string

**max_cost_usd**

number | null format: double

**name**

required

string

**status**

required

Where an experiment is: running until every variant’s run finished.

string

Allowed values: running completed failed cancelled

**trials**

required

integer format: int32

**variants**

required

Array\<object>

One variant of an experiment and how its run went.

object

**cost_per_case_usd**

What one case cost on average, all its trials together.

number | null format: double

**label**

required

string

**model_override**

string | null

**run_elsewhere**

Set when the variant ran outside Everruns and was imported: the system that ran it, such as “mira”.

string | null

**run_id**

required

The variant’s run.

string

**status**

required

Status of an eval run.

string

Allowed values: pending running completed failed cancelled

**summary**

One of:

* object

  Set once the run finished.

  object

  **avg_latency_ms**

  required

  Mean case latency in milliseconds.

  integer format: int64

  **avg_score**

  required

  Mean score across cases, 0.0 to 1.0.

  number format: double

  **avg_turns**

  required

  Mean agent turns per case.

  number format: double

  **cases**

  Distinct cases in the run.

  integer format: int32

  **cases_passed_any_trial**

  Cases that passed in at least one trial (pass\@k).

  integer format: int32

  **cases_passed_every_trial**

  Cases that passed in every trial (pass^k).

  integer format: int32

  **errored**

  required

  Number of cases that errored.

  integer format: int32

  **failed**

  required

  Number of cases that failed.

  integer format: int32

  **pass_rate**

  required

  Fraction of cases that passed, 0.0 to 1.0.

  number format: double

  **passed**

  required

  Number of cases that passed.

  integer format: int32

  **total**

  required

  Total number of cases in the run.

  integer format: int32

  **total_cost_usd**

  What the run spent in USD: the agents’ model calls plus grading. Absent on imported runs, whose cost is in their own metrics.

  number | null format: double

  **total_input_tokens**

  required

  Total input tokens across cases.

  integer format: int64

  **total_output_tokens**

  required

  Total output tokens across cases.

  integer format: int64

  **trials**

  Repetitions per case. `total` counts every repetition.

  integer format: int32

* null

  null

**target**

One of:

* object

  One of:

  * object

    Session creation parameters (mirrors CreateSessionRequest).

    object

    **agent_id**

    Agent to work in this session.

    string | null

    **harness_id**

    Harness for the session. If omitted, org default harness is used.

    string | null

    **harness_name**

    Addressable harness name (alternative to harness_id).

    string | null

    **max_iterations**

    Max LLM iterations per turn.

    integer | null

    **model_id**

    LLM model override.

    string | null

    **system_prompt**

    System prompt override (prepended to agent prompt).

    string | null

    **type**

    required

    string

    Allowed values: session

  * object

    Reference to a deployed app.

    object

    **app_id**

    required

    string

    **type**

    required

    string

    Allowed values: app

  * object

    Label-only target for externally-executed runs (e.g. imported from Mira).

    Carries provider/model labels and opaque params instead of session setup: external runs are ingested already-complete, so everruns never builds a session from this. Mirrors a provider-agnostic `(provider, model)` pair.

    object

    **model**

    required

    string

    **params**

    **provider**

    required

    string

    **type**

    required

    string

    Allowed values: external

* null

  Session creation parameters (mirrors CreateSessionRequest).

  object

  **agent_id**

  Agent to work in this session.

  string | null

  **harness_id**

  Harness for the session. If omitted, org default harness is used.

  string | null

  **harness_name**

  Addressable harness name (alternative to harness_id).

  string | null

  **max_iterations**

  Max LLM iterations per turn.

  integer | null

  **model_id**

  LLM model override.

  string | null

  **system_prompt**

  System prompt override (prepended to agent prompt).

  string | null

  **type**

  required

  string

  Allowed values: session

* object

  Reference to a deployed app.

  object

  **app_id**

  required

  string

  **type**

  required

  string

  Allowed values: app

* object

  Label-only target for externally-executed runs (e.g. imported from Mira).

  Carries provider/model labels and opaque params instead of session setup: external runs are ingested already-complete, so everruns never builds a session from this. Mirrors a provider-agnostic `(provider, model)` pair.

  object

  **model**

  required

  string

  **params**

  **provider**

  required

  string

  **type**

  required

  string

  Allowed values: external

* object

  null

### Example

```json
{
  "baseline": "baseline",
  "comparison": {
    "baseline": "baseline",
    "cases": [
      {
        "case_id": "evalcase_01933b5a000070008000000000000001",
        "case_name": "Refund request, angry customer",
        "cells": [
          {
            "cost_usd": 0.05,
            "label": "candidate",
            "passed": 3,
            "trials": 3
          }
        ]
      }
    ],
    "verdicts": [
      {
        "broke": 1,
        "fixed": 4,
        "interval_high_points": 9,
        "interval_low_points": 1,
        "label": "candidate",
        "paired_cases": 42,
        "pass_rate_delta_points": 5,
        "verdict": "improvement"
      }
    ]
  },
  "dataset_version": 7,
  "eval_id": "eval_01933b5a000070008000000000000001",
  "id": "evalexp_01933b5a000070008000000000000001",
  "max_cost_usd": 10,
  "name": "Luna vs Sol",
  "status": "running",
  "trials": 3,
  "variants": [
    {
      "cost_per_case_usd": 0.031,
      "label": "candidate",
      "model_override": "gpt-6.2-sol",
      "run_elsewhere": "mira",
      "run_id": "evalrun_01933b5a000070008000000000000001",
      "status": "pending",
      "summary": {
        "avg_latency_ms": 8450,
        "avg_score": 0.85,
        "avg_turns": 3.2,
        "cases": 4,
        "cases_passed_any_trial": 4,
        "cases_passed_every_trial": 3,
        "errored": 1,
        "failed": 2,
        "pass_rate": 0.75,
        "passed": 9,
        "total": 12,
        "total_cost_usd": 0.42,
        "total_input_tokens": 14400,
        "total_output_tokens": 10200,
        "trials": 3
      },
      "target": {
        "type": "session"
      }
    }
  ]
}
```

FetchcURL

Request

```js
const url = 'https://app.everruns.com/api/v1/evals/example/experiments';
const options = {
  method: 'POST',
  headers: {'Content-Type': 'application/json'},
  body: '{"max_cost_usd":10,"name":"Luna vs Sol","trials":3,"variants":[{"label":"candidate","model_override":"gpt-6.2-sol","target":{"agent_id":"example","harness_id":"example","harness_name":"example","max_iterations":1,"model_id":"example","system_prompt":"example","type":"session"}}]}'
};


try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

Request

```sh
curl --request POST \
  --url https://app.everruns.com/api/v1/evals/example/experiments \
  --header 'Content-Type: application/json' \
  --data '{ "max_cost_usd": 10, "name": "Luna vs Sol", "trials": 3, "variants": [ { "label": "candidate", "model_override": "gpt-6.2-sol", "target": { "agent_id": "example", "harness_id": "example", "harness_name": "example", "max_iterations": 1, "model_id": "example", "system_prompt": "example", "type": "session" } } ] }'
```

Additional language examples require JavaScript. The default request example is available above.

201

201 · Created

```json
{
  "baseline": "baseline",
  "created_at": "2026-10-12T09:00:00Z",
  "dataset_version": 7,
  "eval_id": "eval_01933b5a000070008000000000000001",
  "id": "evalexp_01933b5a000070008000000000000001",
  "max_cost_usd": 10,
  "name": "Luna vs Sol",
  "status": "running",
  "trials": 3,
  "variants": [
    {
      "cost_per_case_usd": 0.031,
      "label": "candidate",
      "model_override": "gpt-6.2-sol",
      "run_elsewhere": "mira",
      "run_id": "evalrun_01933b5a000070008000000000000001",
      "status": "pending"
    }
  ]
}
```

## Matching endpoints