Check a suggested fix
PreviewRun one of the agent’s evals as an experiment with and without a suggested fix. The fix can be applied once the check shows no regressions.
/v1/evals/fixes/{fix_id}/checkSee authentication for credentials and permissions.
Parameters
Section titled “Parameters”Path Parameters
Section titled “Path Parameters”Prefixed public identifier
Request Bodyrequired
Section titled “Request Bodyrequired”Options for checking a fix.
object
The agent’s eval to check against. Default: the one with most cases.
Spending cap in USD for the whole check.
Trials per case and side, 1 to 20 (default 2).
Example generated
{ "eval_id": "example", "max_cost_usd": 1, "trials": 1}Responses
200Success
Success
A suggested change to an agent’s instructions, written by the Inspector for a finding, and where checking and approving it stand.
object
The text added to the end of the agent’s instructions.
The agent whose instructions the fix changes.
The last check’s result.
object
Cases the fix made fail.
Cases that ran on both sides.
The agent as it is.
object
What the side’s trials cost in USD.
Share of all trials that passed, 0 to 1.
Cases that passed every trial.
The eval’s dataset version at the check.
Name of the eval the check ran.
Cases the fix made pass.
Trials per case and side.
How the variant with the fix compares, noise taken into account.
The agent with the addition.
object
What the side’s trials cost in USD.
Share of all trials that passed, 0 to 1.
Cases that passed every trial.
When the Inspector wrote the fix.
False when the agent’s instructions changed after the last check, so the fix needs checking again before it can be applied.
When a person applied or rejected the fix.
Why it was rejected.
The eval the last check ran, while or after it runs.
The last check’s experiment, while or after it runs.
Prefixed public identifier of the fix.
The finding the fix is for.
proposed (not checked), checking, verified (checked, no
regressions), regressed, check_failed, applied or rejected.
One line on what the change does.
Example
{ "agent_id": "agent_550e8400e29b41d4a716446655440000", "check": { "verdict": "improvement" }, "id": "fix_0190f3a1b2c37d8e9f00112233445566"}