Skip to content
Docs
GitHub

Check a suggested fix

Preview

Run one of the agent’s evals as an experiment with and without a suggested fix. The fix can be applied once the check shows no regressions.

Public APIRequires the evals feature flag

POST/v1/evals/fixes/{fix_id}/check

See authentication for credentials and permissions.

fix_id
required
string

Prefixed public identifier

Media typeapplication/json

Options for checking a fix.

object
eval_id

The agent’s eval to check against. Default: the one with most cases.

string | null
max_cost_usd

Spending cap in USD for the whole check.

number | null format: double
trials

Trials per case and side, 1 to 20 (default 2).

integer | null format: int32

Example generated

{
"eval_id": "example",
"max_cost_usd": 1,
"trials": 1
}

Responses

200Success

Success

Media typeapplication/json

A suggested change to an agent’s instructions, written by the Inspector for a finding, and where checking and approving it stand.

object
addition
required

The text added to the end of the agent’s instructions.

string
agent_id
required

The agent whose instructions the fix changes.

string
check
One of:

The last check’s result.

object
broke
required

Cases the fix made fail.

integer format: int32
cases
required

Cases that ran on both sides.

integer format: int32
current
required

The agent as it is.

object
cost_usd
required

What the side’s trials cost in USD.

number format: double
pass_rate
required

Share of all trials that passed, 0 to 1.

number format: double
passed
required

Cases that passed every trial.

integer format: int32
dataset_version
required

The eval’s dataset version at the check.

integer format: int32
eval_name
required

Name of the eval the check ran.

string
fixed
required

Cases the fix made pass.

integer format: int32
trials
required

Trials per case and side.

integer format: int32
verdict
required

How the variant with the fix compares, noise taken into account.

string
Allowed values: improvement regression within_noise not_enough_data
with_fix
required

The agent with the addition.

object
cost_usd
required

What the side’s trials cost in USD.

number format: double
pass_rate
required

Share of all trials that passed, 0 to 1.

number format: double
passed
required

Cases that passed every trial.

integer format: int32
created_at
required

When the Inspector wrote the fix.

string format: date-time
current
required

False when the agent’s instructions changed after the last check, so the fix needs checking again before it can be applied.

boolean
decided_at

When a person applied or rejected the fix.

string | null format: date-time
decision_note

Why it was rejected.

string | null
eval_id

The eval the last check ran, while or after it runs.

string | null
experiment_id

The last check’s experiment, while or after it runs.

string | null
id
required

Prefixed public identifier of the fix.

string
issue_id
required

The finding the fix is for.

string format: uuid
status
required

proposed (not checked), checking, verified (checked, no regressions), regressed, check_failed, applied or rejected.

string
summary
required

One line on what the change does.

string

Example

{
"agent_id": "agent_550e8400e29b41d4a716446655440000",
"check": {
"verdict": "improvement"
},
"id": "fix_0190f3a1b2c37d8e9f00112233445566"
}