miraEval framework · Open source
Your evals belong in your repo.
Mira is a code-first eval framework for agents and tools — built for multi-turn, tool-using, long-running trajectories. Named after the Ukrainian міра — measure — because that is what an eval framework is for. Datasets, the study, and every saved run are just files you commit.
mira — ~/acme-agentThe real mira run output shape. Flip a flag and the grid re-plans: the two -- flags narrow what runs, and the third is an environment variable — without it that target is skipped, not failed.
The core model
A dataset, a subject, some scorers — crossed with your matrix.
Four entities and one line of algebra. Everything else in Mira — selection, resume, reporting — is the host doing bookkeeping on top of them.
Eval = Dataset(Sample…) + Subject + [Scorer…] × model matrix × axesSample
the inputOne dataset row: input turns, an optional expected, seeded files, tags, and metadata. Language-agnostic JSON — write it inline for small evals, or load Dataset::jsonl for larger sets.
Subject
the thing under testOne adapter per shape: an in-process closure, an external binary, or a live runtime session. Every one produces the same normalized Transcript, so scoring and reporting are shared.
Scorer
the verdictscore(&Sample, &Transcript) -> Score — a value in 0..1, a pass, and a reason. Deterministic built-ins, operational budgets, closures, and LLM-as-judge all share the one trait.
Target
the axisOne matrix case, and the first-class axis: a (label, provider, model, available) tuple with no API keys and no SDK types. For an agent eval the target is the agent — Target::cli("yolop").
The seam
Your evals and the runner are two processes.
Eval definitions and the runner sit on opposite sides of a process boundary, talking newline-delimited JSON over stdio. That single decision is why a study can be written in any language — and why your provider keys never leave your own program.
It owns subjects and scoring, and knows nothing about selection, matrices, saved runs, or rendering. Provider API keys live only here and never cross the wire.
mira CLI.It spawns the study, enumerates evals, plans selection × matrix, drives execution with bounded per-provider concurrency, then aggregates, saves the run, and renders.
initialize, list, and run, plus fire-and-forget event/log notifications and capability-gated extensions. Every payload tolerates unknown fields, so a newer study and an older host interoperate.
Selection only ever subsets the grid the study declared — the host never adds cases.— docs/how-it-works.md
Scoring
One open vocabulary, freely composed.
Scorers grade three surfaces: the final response, the structured ATIF trajectory, and the operational metrics. A case passes only if every applicable score passes — so a cost regression fails as loudly as a wrong answer.
Text & output
the final responsesucceedednon_emptycontainsnot_containsequalsregexmatches_expectedjson_validjson_field_equals
Tools
what was calledtool_calledtool_not_calledtool_calls_withintools_used_exactlytool_called_before
Trajectory
arguments & observationstool_called_withtool_arg_matchesobservation_containssteps_within
Budgets
tokens, cost, latencytokens_withinoutput_tokens_withincost_withinturns_withinlatency_withinttft_withinmetric_withinmetric_at_least
Files
the captured workspacefile_existsfile_contains
Compose
anything aboveall_ofany_ofnotscorermodel_graded
A scorer that cannot run — no API key, a rate limit, a 5xx — returns N/A instead of a misleading fail: excluded from the verdict and the aggregate, rendered –, never counted as a JUnit failure. That is how a run with no credentials still comes back green.
Subjects
Three shapes, one transcript.
A subject is whatever you point the eval at. Pick the adapter that matches its shape — the rest of the framework does not change.
Write it anywhere
A study is a program that speaks the protocol.
The contract is the protocol, not a list of blessed languages: any program that speaks newline-delimited JSON over stdio is a first-class study. Rust is the native path, and the Python and TypeScript SDKs are not bindings — they are standalone libraries generated from the same JSON Schema the Rust host is, so they cannot drift from the wire format.
A single-file Rust study needs no Cargo.toml — dependencies go in cargo-script frontmatter and mira run --study study.rs resolves the runner from the extension. The Python SDK is pure stdlib; the TypeScript SDK has zero runtime dependencies.
use mira::scorer::{contains, succeeded, cost_within};
use mira::subject::subject_fn;
use mira::{eval, Eval, Transcript};
#[eval]
fn greet() -> Eval {
Eval::new("greet")
.sample("hi", "Say hi and tell me the answer to life.")
.subject(subject_fn(|sample, cx| async move {
Transcript::response(call_model(cx.target(), sample).await)
}))
.scorer(succeeded())
.scorer(contains("42"))
.scorer(cost_within(0.01))
.build()
}
#[tokio::main]
async fn main() -> std::io::Result<()> {
mira::Study::registered().serve().await
}import mira
study = mira.Study("my-evals", version="0.1.0")
@study.eval(
name="greet",
samples=[mira.Sample("hi", prompt="Say hi and the answer.")],
targets=[mira.target("sim")],
scorers=[mira.succeeded(), mira.contains("42")],
)
def greet(sample, cx):
response = call_model(cx.target, sample.text)
return mira.transcript(response, usage=mira.Usage(output_tokens=8))
if __name__ == "__main__":
study.serve()import { Study, sample, target, succeeded, contains, transcript }
from "mira-eval";
const study = new Study("my-evals", { version: "0.1.0" });
study.eval({
name: "greet",
samples: [sample("hi", { prompt: "Say hi and the answer." })],
targets: [target("sim")],
scorers: [succeeded(), contains("42")],
run: async (s, cx) => transcript(await callModel(cx.target, s.text)),
});
study.serve();Results are files
Every run is a folder you can commit.
There is no server to stand up and no third party holding your results. mira run saves a run folder by default — plain JSON your coding agent can read, diff, and chart without an SDK or a query language.
Each run records the git commit, branch, and dirty flag, the box (os, arch, cpus, mem), the mira version, and auto-detected CI labels — so a number from three weeks ago is still interpretable today.
--resume <run_id> reopens a run folder, skips the cases already recorded, and runs only what is missing. A fresh run mints a new id and reuses nothing — no silent reuse of stale results.
mira report <run_id> re-renders from the stored case files with no study process spawned, and mira export <run_id> --format atif emits standalone ATIF trajectory documents.
meta.jsonRun identity: id, study, timestamps, summary, and the environment it came from.report.jsonThe canonical machine-readable record — summary plus every case.report.htmlA self-contained, dependency-free transcript viewer you can open from a CI artifact.cases/<key>/result.jsonOne finished case (eval/sample@target[…]#trial), written atomically as it completes.
In the loop
Drops straight into CI.
Non-zero exit when a case fails, one flag per artifact, and a progress bar that hides itself under a non-TTY so it never pollutes logs.
--format jsoncanonical record--format jsonlone case per line--format csvlong-format, for analysis--format junitCI test reporting--format mdpull-request comments--format htmlself-contained viewer--trials N runs each case N times for pass@k and standard deviation, with --seed for reproducibility.--provider-concurrency anthropic=2,openai=4 caps each provider, with adaptive throttling on rate limits.mira run --experiment plan.toml, without touching the durable study.mira doctor lints mira.toml, the study listing, and the saved runs in one pass; --fix applies the safe repairs. Errors exit non-zero, so it can gate CI.Quick start
From nothing to a scored matrix.
Homebrew on macOS & Linux, or grab the release binary with cargo binstall mira-cli.
One file. #[eval] discovers it the way cargo test discovers tests — no registration, no config.
The run saves itself to ./results/<run_id>/. Commit the folder and point your agent at it.
Measure the agent you already have.
Mira is open source and MIT-licensed. Install the host, write one file, and keep the results in the repo where the code lives.