Skip to content
Docs
GitHub

Check an experiment's variant against a bar, for CI

Preview

Check an experiment’s variant against a bar, for CI. Answers 409 while runs are going and 422 when the variant misses the bar, so the command exits non-zero.

Public APIRequires the evals feature flag

GET/v1/evals/{eval_id}/experiments/{experiment_id}/gate

See authentication for credentials and permissions.

eval_id
required
string

Prefixed public identifier

experiment_id
required
string

Prefixed public identifier

Responses

200Success

Success

Media typeapplication/json

Result of an experiment gate that passed.

object
pass_rate
required
number format: double
passed
required
boolean
passed_every_trial
required

Share of cases passed in every trial (pass^k).

number format: double
variant
required

The variant the gate judged.

string
verdict
One of:

How a variant compares with the baseline.

string
Allowed values: improvement regression within_noise not_enough_data

Example

{
"pass_rate": 0.93,
"passed_every_trial": 0.9,
"variant": "candidate",
"verdict": "improvement"
}
404Experiment not found

Experiment not found

409Some variant's run is still going

Some variant’s run is still going

422The variant missed the bar

The variant missed the bar