# Mira

**Navigation:** [Products](/products/) | [Use Cases](/use-cases/) | [Blog](/blog/) | [Docs](https://docs.everruns.com/) | [GitHub](https://github.com/everruns/mira)

---

> Your evals belong in your repo.

Mira is a code-first eval framework for agents and tools, built for multi-turn, tool-using, long-running trajectories. Named after the Ukrainian *міра*, measure, because that is what an eval framework is for. Datasets, the study, and every saved run are just files you commit.

```bash
brew install everruns/tap/mira
```

- **30+ scorers**: built in, composable
- **Any language**: Rust, Python, TS SDKs
- **MIT**: licensed, in Rust

---

## A dataset, a subject, some scorers, crossed with your matrix.

Four entities and one line of algebra. Everything else in Mira, selection, resume, reporting, is the host doing bookkeeping on top of them.

```text
Eval = Dataset(Sample…) + Subject + [Scorer…]  ×  model matrix × axes
```

### `Sample`: the input

One dataset row: input turns, an optional `expected`, seeded `files`, `tags`, and `metadata`. Language-agnostic JSON, write it inline for small evals, or load `Dataset::jsonl` for larger sets.

### `Subject`: the thing under test

One adapter per *shape*: an in-process closure, an external binary, or a live runtime session. Every one produces the same normalized `Transcript`, so scoring and reporting are shared.

### `Scorer`: the verdict

`score(&Sample, &Transcript) -> Score`, a value in `0..1`, a pass, and a reason. Deterministic built-ins, operational budgets, closures, and LLM-as-judge all share the one trait.

### `Target`: the axis

One matrix case, and the first-class axis: a `(label, provider, model, available)` tuple with no API keys and no SDK types. For an agent eval the target *is* the agent, `Target::cli("yolop")`.

---

## Your evals and the runner are two processes.

Eval definitions and the runner sit on opposite sides of a process boundary, talking newline-delimited JSON over stdio. That single decision is why a study can be written in any language, and why your provider keys never leave your own program.

- **Study: Your eval program.** It owns subjects and scoring, and knows nothing about selection, matrices, saved runs, or rendering. Provider API keys live only here and never cross the wire.
- **Host: The `mira` CLI.** It spawns the study, enumerates evals, plans selection × matrix, drives execution with bounded per-provider concurrency, then aggregates, saves the run, and renders.
- **Protocol: Three methods, versioned.** `initialize`, `list`, and `run`, plus fire-and-forget `event`/`log` notifications and capability-gated extensions. Every payload tolerates unknown fields, so a newer study and an older host interoperate.

> Selection only ever subsets the grid the study declared, the host never adds cases.
>
> - docs/how-it-works.md

---

## One open vocabulary, freely composed.

Scorers grade three surfaces: the final response, the structured [ATIF](https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md) trajectory, and the operational metrics. A case passes only if every applicable score passes, so a cost regression fails as loudly as a wrong answer.

- **Text & output** (the final response): `succeeded`, `non_empty`, `contains`, `not_contains`, `equals`, `regex`, `matches_expected`, `json_valid`, `json_field_equals`
- **Tools** (what was called): `tool_called`, `tool_not_called`, `tool_calls_within`, `tools_used_exactly`, `tool_called_before`
- **Trajectory** (arguments & observations): `tool_called_with`, `tool_arg_matches`, `observation_contains`, `steps_within`
- **Budgets** (tokens, cost, latency): `tokens_within`, `output_tokens_within`, `cost_within`, `turns_within`, `latency_within`, `ttft_within`, `metric_within`, `metric_at_least`
- **Files** (the captured workspace): `file_exists`, `file_contains`
- **Compose** (anything above): `all_of`, `any_of`, `not`, `scorer`, `model_graded`

A scorer that cannot run, no API key, a rate limit, a 5xx, returns **N/A** instead of a misleading fail: excluded from the verdict and the aggregate, rendered `–`, never counted as a JUnit failure. That is how a run with no credentials still comes back green.

---

## Three shapes, one transcript.

A subject is whatever you point the eval at. Pick the adapter that matches its shape, the rest of the framework does not change.

### In-process

A closure. The shortest path for prompt, library, and model evals, no process boundary, no serialization.

`subject_fn(|sample, cx| …)`

### Any binary

The polyglot path: evaluate an external program in any language. It reads stdout, an ATIF trajectory file, or a canonical JSONL transcript, and can seed and capture files.

`CliSubject::new("./agent")`

### Live runtime

Drives a real [Everruns Runtime](/products/runtime) session, so long-running multi-turn agents are evaluated as they actually run.

`mira_everruns::RuntimeSubject`

---

## A study is a program that speaks the protocol.

The contract is the protocol, not a list of blessed languages: any program that speaks newline-delimited JSON over stdio is a first-class study. Rust is the native path, and the Python and TypeScript SDKs are not bindings, they are standalone libraries generated from the same JSON Schema the Rust host is, so they cannot drift from the wire format.

### Rust (`study.rs`)

```rust
use mira::scorer::{contains, succeeded, cost_within};
use mira::subject::subject_fn;
use mira::{eval, Eval, Transcript};

#[eval]
fn greet() -> Eval {
    Eval::new("greet")
        .sample("hi", "Say hi and tell me the answer to life.")
        .subject(subject_fn(|sample, cx| async move {
            Transcript::response(call_model(cx.target(), sample).await)
        }))
        .scorer(succeeded())
        .scorer(contains("42"))
        .scorer(cost_within(0.01))
        .build()
}

#[tokio::main]
async fn main() -> std::io::Result<()> {
    mira::Study::registered().serve().await
}
```

```bash
mira run --study study.rs
```

### Python (`study.py`)

```python
import mira

study = mira.Study("my-evals", version="0.1.0")

@study.eval(
    name="greet",
    samples=[mira.Sample("hi", prompt="Say hi and the answer.")],
    targets=[mira.target("sim")],
    scorers=[mira.succeeded(), mira.contains("42")],
)
def greet(sample, cx):
    response = call_model(cx.target, sample.text)
    return mira.transcript(response, usage=mira.Usage(output_tokens=8))

if __name__ == "__main__":
    study.serve()
```

```bash
mira run --study study.py
```

### TypeScript (`study.mjs`)

```typescript
import { Study, sample, target, succeeded, contains, transcript }
  from "mira-eval";

const study = new Study("my-evals", { version: "0.1.0" });

study.eval({
  name: "greet",
  samples: [sample("hi", { prompt: "Say hi and the answer." })],
  targets: [target("sim")],
  scorers: [succeeded(), contains("42")],
  run: async (s, cx) => transcript(await callModel(cx.target, s.text)),
});

study.serve();
```

```bash
mira run --study-cmd "node study.mjs"
```

A single-file Rust study needs no `Cargo.toml`, dependencies go in cargo-script frontmatter and `mira run --study study.rs` resolves the runner from the extension. The Python SDK is pure stdlib; the TypeScript SDK has zero runtime dependencies.

---

## Every run is a folder you can commit.

There is no server to stand up and no third party holding your results. `mira run` saves a run folder by default, plain JSON your coding agent can read, diff, and chart without an SDK or a query language.

`results/<run_id>/`

- `meta.json`: Run identity: id, study, timestamps, summary, and the environment it came from.
- `report.json`: The canonical machine-readable record, summary plus every case.
- `report.html`: A self-contained, dependency-free transcript viewer you can open from a CI artifact.
- `cases/<key>/result.json`: One finished case (`eval/sample@target[…]#trial`), written atomically as it completes.

### Stamped with where it came from.

*Provenance.* Each run records the git commit, branch, and dirty flag, the box (`os`, `arch`, `cpus`, `mem`), the `mira` version, and auto-detected CI labels, so a number from three weeks ago is still interpretable today.

### Finish a run, do not restart it.

*Resume.* `--resume <run_id>` reopens a run folder, skips the cases already recorded, and runs only what is missing. A fresh run mints a new id and reuses nothing, no silent reuse of stale results.

### Reports without re-execution.

*Re-render.* `mira report <run_id>` re-renders from the stored case files with no study process spawned, and `mira export <run_id> --format atif` emits standalone ATIF trajectory documents.

---

## Drops straight into CI.

Non-zero exit when a case fails, one flag per artifact, and a progress bar that hides itself under a non-TTY so it never pollutes logs.

- `--format json`: canonical record
- `--format jsonl`: one case per line
- `--format csv`: long-format, for analysis
- `--format junit`: CI test reporting
- `--format md`: pull-request comments
- `--format html`: self-contained viewer

- **Green offline.** A target whose API key is missing is marked unavailable and *skipped*, not failed, so a fresh checkout runs clean before anyone hands out credentials.
- **Repetitions and variance.** `--trials N` runs each case N times for pass@k and standard deviation, with `--seed` for reproducibility.
- **Bounded concurrency.** `--provider-concurrency anthropic=2,openai=4` caps each provider, with adaptive throttling on rate limits.
- **Experiment plans.** Compare short-lived treatments, prompt candidates, two binaries, a dependency bump, from an external TOML plan with `mira run --experiment plan.toml`, without touching the durable study.
- **Diagnose the setup.** `mira doctor` lints `mira.toml`, the study listing, and the saved runs in one pass; `--fix` applies the safe repairs. Errors exit non-zero, so it can gate CI.
- **Agent-authored.** Mira ships an agent skill, so your coding agent already knows the API, the CLI, and the conventions, describe the eval you want and let it write the study.

---

## From nothing to a scored matrix.

1. **Install the host.** Homebrew on macOS & Linux, or grab the release binary with `cargo binstall mira-cli`.
2. **Write a study.** One file. `#[eval]` discovers it the way `cargo test` discovers tests, no registration, no config.
3. **Run the matrix.** The run saves itself to `./results/<run_id>/`. Commit the folder and point your agent at it.

```bash
$ brew install everruns/tap/mira
$ mira list --study study.rs
$ mira run  --study study.rs
# subset it like cargo test:
$ mira run  --study study.rs --tag smoke
# self-contained report for a CI artifact:
$ mira run  --study study.rs --format html --out report.html
✓ 18 passed / 20 scored (2 failed, 0 n/a, 6 skipped)
```

---

## Measure the agent you already have.

Mira is open source and MIT-licensed. Install the host, write one file, and keep the results in the repo where the code lives.

Install: `brew install everruns/tap/mira`

GitHub: https://github.com/everruns/mira