# Mira

**Navigation:** [Products](/products/) | [Use Cases](/use-cases/) | [Blog](/blog/) | [Docs](https://docs.everruns.com/) | [GitHub](https://github.com/everruns/mira)

---

> Your evals belong in your repo.

Mira is a code-first eval framework for agents and tools — built for multi-turn, tool-using, long-running trajectories. Named after the Ukrainian *міра* — measure — because that is what an eval framework is for. Datasets, the study, and every saved run are just files you commit.

```bash
brew install everruns/tap/mira
```

- **30+ scorers** — built in, composable
- **Any language** — Rust, Python, TS SDKs
- **MIT** — licensed, in Rust

---

## A dataset, a subject, some scorers — crossed with your matrix.

Four entities and one line of algebra. Everything else in Mira — selection, resume, reporting — is the host doing bookkeeping on top of them.

```text
Eval = Dataset(Sample…) + Subject + [Scorer…]  ×  model matrix × axes
```

### `Sample` — the input

One dataset row: input turns, an optional `expected`, seeded `files`, `tags`, and `metadata`. Language-agnostic JSON — write it inline for small evals, or load `Dataset::jsonl` for larger sets.

### `Subject` — the thing under test

One adapter per *shape*: an in-process closure, an external binary, or a live runtime session. Every one produces the same normalized `Transcript`, so scoring and reporting are shared.

### `Scorer` — the verdict

`score(&Sample, &Transcript) -> Score` — a value in `0..1`, a pass, and a reason. Deterministic built-ins, operational budgets, closures, and LLM-as-judge all share the one trait.

### `Target` — the axis

One matrix case, and the first-class axis: a `(label, provider, model, available)` tuple with no API keys and no SDK types. For an agent eval the target *is* the agent — `Target::cli("yolop")`.

---

## Your evals and the runner are two processes.

Eval definitions and the runner sit on opposite sides of a process boundary, talking newline-delimited JSON over stdio. That single decision is why a study can be written in any language — and why your provider keys never leave your own program.

- **Study — Your eval program.** It owns subjects and scoring, and knows nothing about selection, matrices, saved runs, or rendering. Provider API keys live only here and never cross the wire.
- **Host — The `mira` CLI.** It spawns the study, enumerates evals, plans selection × matrix, drives execution with bounded per-provider concurrency, then aggregates, saves the run, and renders.
- **Protocol — Three methods, versioned.** `initialize`, `list`, and `run`, plus fire-and-forget `event`/`log` notifications and capability-gated extensions. Every payload tolerates unknown fields, so a newer study and an older host interoperate.

> Selection only ever subsets the grid the study declared — the host never adds cases.
>
> — docs/how-it-works.md

---

## One open vocabulary, freely composed.

Scorers grade three surfaces: the final response, the structured [ATIF](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trajectory, and the operational metrics. A case passes only if every applicable score passes — so a cost regression fails as loudly as a wrong answer.

- **Text & output** (the final response): `succeeded`, `non_empty`, `contains`, `not_contains`, `equals`, `regex`, `matches_expected`, `json_valid`, `json_field_equals`
- **Tools** (what was called): `tool_called`, `tool_not_called`, `tool_calls_within`, `tools_used_exactly`, `tool_called_before`
- **Trajectory** (arguments & observations): `tool_called_with`, `tool_arg_matches`, `observation_contains`, `steps_within`
- **Budgets** (tokens, cost, latency): `tokens_within`, `output_tokens_within`, `cost_within`, `turns_within`, `latency_within`, `ttft_within`, `metric_within`, `metric_at_least`
- **Files** (the captured workspace): `file_exists`, `file_contains`
- **Compose** (anything above): `all_of`, `any_of`, `not`, `scorer`, `model_graded`

A scorer that cannot run — no API key, a rate limit, a 5xx — returns **N/A** instead of a misleading fail: excluded from the verdict and the aggregate, rendered `–`, never counted as a JUnit failure. That is how a run with no credentials still comes back green.

---

## Three shapes, one transcript.

A subject is whatever you point the eval at. Pick the adapter that matches its shape — the rest of the framework does not change.

### In-process

A closure. The shortest path for prompt, library, and model evals — no process boundary, no serialization.

`subject_fn(|sample, cx| …)`

### Any binary

The polyglot path: evaluate an external program in any language. It reads stdout, an ATIF trajectory file, or a canonical JSONL transcript, and can seed and capture files.

`CliSubject::new("./agent")`

### Live runtime

Drives a real [Everruns Runtime](/products/runtime) session, so long-running multi-turn agents are evaluated as they actually run.

`mira_everruns::RuntimeSubject`

---

## A study is a program that speaks the protocol.

The contract is the protocol, not a list of blessed languages: any program that speaks newline-delimited JSON over stdio is a first-class study. Rust is the native path, and the Python and TypeScript SDKs are not bindings — they are standalone libraries generated from the same JSON Schema the Rust host is, so they cannot drift from the wire format.

### Rust (`study.rs`)

```rust
use mira::scorer::{contains, succeeded, cost_within};
use mira::subject::subject_fn;
use mira::{eval, Eval, Transcript};

#[eval]
fn greet() -> Eval {
    Eval::new("greet")
        .sample("hi", "Say hi and tell me the answer to life.")
        .subject(subject_fn(|sample, cx| async move {
            Transcript::response(call_model(cx.target(), sample).await)
        }))
        .scorer(succeeded())
        .scorer(contains("42"))
        .scorer(cost_within(0.01))
        .build()
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    mira::Study::registered().serve().await
}
```

```bash
mira run --study study.rs
```

### Python (`study.py`)

```python
import mira

study = mira.Study("my-evals", version="0.1.0")

@study.eval(
    name="greet",
    samples=[mira.Sample("hi", prompt="Say hi and the answer.")],
    targets=[mira.target("sim")],
    scorers=[mira.succeeded(), mira.contains("42")],
)
def greet(sample, cx):
    response = call_model(cx.target, sample.text)
    return mira.transcript(response, usage=mira.Usage(output_tokens=8))

if __name__ == "__main__":
    study.serve()
```

```bash
mira run --study study.py
```

### TypeScript (`study.mjs`)

```typescript
import { Study, sample, target, succeeded, contains, transcript }
  from "mira-eval";

const study = new Study("my-evals", { version: "0.1.0" });

study.eval({
  name: "greet",
  samples: [sample("hi", { prompt: "Say hi and the answer." })],
  targets: [target("sim")],
  scorers: [succeeded(), contains("42")],
  run: async (s, cx) => transcript(await callModel(cx.target, s.text)),
});

study.serve();
```

```bash
mira run --study-cmd "node study.mjs"
```

A single-file Rust study needs no `Cargo.toml` — dependencies go in cargo-script frontmatter and `mira run --study study.rs` resolves the runner from the extension. The Python SDK is pure stdlib; the TypeScript SDK has zero runtime dependencies.

---

## Every run is a folder you can commit.

There is no server to stand up and no third party holding your results. `mira run` saves a run folder by default — plain JSON your coding agent can read, diff, and chart without an SDK or a query language.

`results/<run_id>/`

- `meta.json` — Run identity: id, study, timestamps, summary, and the environment it came from.
- `report.json` — The canonical machine-readable record — summary plus every case.
- `report.html` — A self-contained, dependency-free transcript viewer you can open from a CI artifact.
- `cases/<key>/result.json` — One finished case (`eval/sample@target[…]#trial`), written atomically as it completes.

### Stamped with where it came from.

*Provenance.* Each run records the git commit, branch, and dirty flag, the box (`os`, `arch`, `cpus`, `mem`), the `mira` version, and auto-detected CI labels — so a number from three weeks ago is still interpretable today.

### Finish a run, do not restart it.

*Resume.* `--resume <run_id>` reopens a run folder, skips the cases already recorded, and runs only what is missing. A fresh run mints a new id and reuses nothing — no silent reuse of stale results.

### Reports without re-execution.

*Re-render.* `mira report <run_id>` re-renders from the stored case files with no study process spawned, and `mira export <run_id> --format atif` emits standalone ATIF trajectory documents.

---

## Drops straight into CI.

Non-zero exit when a case fails, one flag per artifact, and a progress bar that hides itself under a non-TTY so it never pollutes logs.

- `--format json` — canonical record
- `--format jsonl` — one case per line
- `--format csv` — long-format, for analysis
- `--format junit` — CI test reporting
- `--format md` — pull-request comments
- `--format html` — self-contained viewer

- **Green offline.** A target whose API key is missing is marked unavailable and *skipped*, not failed, so a fresh checkout runs clean before anyone hands out credentials.
- **Repetitions and variance.** `--trials N` runs each case N times for pass@k and standard deviation, with `--seed` for reproducibility.
- **Bounded concurrency.** `--provider-concurrency anthropic=2,openai=4` caps each provider, with adaptive throttling on rate limits.
- **Experiment plans.** Compare short-lived treatments — prompt candidates, two binaries, a dependency bump — from an external TOML plan with `mira run --experiment plan.toml`, without touching the durable study.
- **Diagnose the setup.** `mira doctor` lints `mira.toml`, the study listing, and the saved runs in one pass; `--fix` applies the safe repairs. Errors exit non-zero, so it can gate CI.
- **Agent-authored.** Mira ships an agent skill, so your coding agent already knows the API, the CLI, and the conventions — describe the eval you want and let it write the study.

---

## From nothing to a scored matrix.

1. **Install the host.** Homebrew on macOS & Linux, or grab the release binary with `cargo binstall mira-cli`.
2. **Write a study.** One file. `#[eval]` discovers it the way `cargo test` discovers tests — no registration, no config.
3. **Run the matrix.** The run saves itself to `./results/<run_id>/`. Commit the folder and point your agent at it.

```bash
$ brew install everruns/tap/mira
$ mira list --study study.rs
$ mira run  --study study.rs
# subset it like cargo test:
$ mira run  --study study.rs --tag smoke
# self-contained report for a CI artifact:
$ mira run  --study study.rs --format html --out report.html
✓ 18 passed / 20 scored (2 failed, 0 n/a, 6 skipped)
```

---

## Measure the agent you already have.

Mira is open source and MIT-licensed. Install the host, write one file, and keep the results in the repo where the code lives.

Install: `brew install everruns/tap/mira`

GitHub: https://github.com/everruns/mira