# Environment Variables
> Every Everruns environment variable: database connections, authentication, encryption, and development mode.
Source:
## Summary
A production control plane (any `DEPLOYMENT_GRADE` other than `dev`, `DEV_MODE` unset) refuses to start without the first three variables. Everything else has a working default; each row links to the section that documents it.
| Variable | Process | Required | Without it |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------- |
| [`DATABASE_URL`](#database_url) | API | Yes, unless `DEV_MODE=true` | Startup fails. Dev mode ignores it and starts its own temporary PostgreSQL. |
| [`AUTH_MODE`](#other-server-variables) | API | Yes, unless the grade is `dev` | Startup fails: unset means `none`, which is allowed only at `dev` grade. |
| [`WORKER_GRPC_AUTH_TOKEN`](#worker_grpc_auth_token) | API and workers (same value) | Yes, unless `DEV_MODE=true` | The API fails to start its gRPC server; workers cannot authenticate. |
| [`SECRETS_ENCRYPTION_KEY`](#llm-provider-api-keys) | API | Recommended | The API starts, but cannot store provider keys or other encrypted values. |
| [`DEV_MODE`](#dev_mode), [`DEPLOYMENT_GRADE`](#deployment_grade) | API and workers | No | Production behavior (`prod` grade). |
| [`PUBLIC_APP_URL`](#public_app_url), [`AUTH_LOGIN_ORIGIN`](#auth_login_origin), [`API_PREFIX`](#api_prefix), [`CORS_ALLOWED_ORIGINS`](#cors_allowed_origins), [`HTTP_ADDR`](#http_addr) | API | No | `http://localhost:9300`, same-origin login, `/api`, CORS off, `0.0.0.0:9000`. |
| [`DATABASE_UNPOOLED_URL`](#database_unpooled_url), [`DATABASE_POOL_MAX`](#other-server-variables), [`DATABASE_POOL_MIN`](#other-server-variables) | API | No | Listeners reuse `DATABASE_URL`; pool of 5 to 50 connections. |
| [`VALKEY_URL`](#valkey_url) | API | No | Rate limits are per instance. |
| [`NATS_URL`](#nats_url) | API and workers | No | PostgreSQL `NOTIFY` and in-memory broadcast. |
| [`STORAGE_*`](#object-storage-s3-compatible-blob-backend) | API | No | File and image bytes stay in PostgreSQL. |
| [`SERVER_GRPC_ADDRESS`](#server_grpc_address), [`SERVER_GRPC_BIND_ADDR`](#server_grpc_bind_addr), [`WORKER_GRPC_CONNECT_TIMEOUT`](#worker_grpc_connect_timeout), [`WORKER_GRPC_TLS_*`](#worker_grpc_tls_cert) | API or workers | No | Plain gRPC on `127.0.0.1:9001` / `0.0.0.0:9001`. |
| [`DEFAULT_*_API_KEY`](#default-api-keys-development-convenience) | API | No | Providers need keys configured in Models > Providers. |
| [`UTILITY_*`](#system-model-keys) | API and workers | No | Analyze, Health, and model-backed guardrail checks are unavailable or skipped. |
| [`EMAIL_PROVIDER`](#system-email-delivery), [`RESEND_*`](#system-email-delivery) | API | When sending email | Email delivery is disabled. |
| [`RATE_LIMIT_API_REQUESTS_PER_MINUTE`](#other-server-variables), [`TRUSTED_PROXY_HOPS`](#other-server-variables), [`EXPECTED_INSTANCES`](#other-server-variables), [`EVENT_RETENTION_DAYS`](#other-server-variables), [`SSE_*`](#sse-streaming-configuration) | API | No | 1200 requests per minute, one trusted proxy, one instance, no event archiving, default SSE limits. |
| [`OTEL_*`](#opentelemetry-configuration), [`EVERRUNS_TRACE_CONVENTIONS`](#everruns_trace_conventions), [`BRAINTRUST_*`](#braintrust-integration) | API and workers | No | Tracing and Braintrust export are off. |
## DEV\_MODE
Enable development mode. No PostgreSQL to install or run: the server starts a temporary PostgreSQL of its own and deletes it on exit.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | `false` |
**Example:**
```bash
# Start in dev mode (no database to set up)
DEV_MODE=true ./target/debug/everruns-server
# Or with 1
DEV_MODE=1 ./target/debug/everruns-server
```
**Notes:**
* When enabled, the server downloads PostgreSQL 17 binaries on first use (cached under `~/.cache/everruns/postgres`, or `EVERRUNS_PG_CACHE`), starts a cluster on a free loopback port, and runs migrations into a fresh database. `DATABASE_URL` is ignored.
* Data does not survive a restart. Use `just start-all` for a persistent local database.
* When run as root, the cluster runs as `nobody`, because PostgreSQL refuses to run as root.
* All data is lost when the server stops
* gRPC server and worker communication are disabled
* Stale task reclamation is disabled
* Useful for quick local development and testing
* Not suitable for production or multi-instance deployments
**Limitations in dev mode:**
* No persistence (data is lost on restart)
* No worker support (all execution happens in-process)
* No distributed tracing of worker activities
* Single-instance only
## DEPLOYMENT\_GRADE
Deployment environment grade. Controls which features and capabilities are available.
| Property | Value |
| ------------ | ------------------------------------ |
| **Required** | No |
| **Default** | `prod` (or `dev` if `DEV_MODE=true`) |
**Valid values:**
| Grade | Description |
| --------- | ----------------------------------------------- |
| `dev` | Development - all experimental features enabled |
| `poc` | Proof of concept / demo environment |
| `preview` | Preview/staging environment |
| `prod` | Production - only stable features |
**Example:**
```bash
# Run in development mode with experimental features
DEPLOYMENT_GRADE=dev ./target/debug/everruns-server
# Production mode (default)
DEPLOYMENT_GRADE=prod ./target/debug/everruns-server
```
**Notes:**
* If not set, falls back to `DEV_MODE`: if `DEV_MODE=true`, uses `dev`; otherwise uses `prod`
* Experimental capabilities (e.g., Docker Container) are only available in `dev` grade
* Experimental seed agents (e.g., Python Coder) are only created in `dev` grade
* Use `dev` for local development and testing experimental features
* Use `prod` for production deployments
## Feature rollout grades
Each `FEATURE_` variable overrides that feature’s rollout grade. Feature grades are separate from `DEPLOYMENT_GRADE`, which describes the running deployment.
| Value | Behaviour |
| ---------- | ------------------------------------------------------------------------- |
| `dev` | Available only on a local dev deployment; enabled there by default |
| `internal` | Disabled until a platform operator enables it for a specific organisation |
| `adoption` | Disabled by default; an organisation owner/admin may enable it |
| `prod` | Enabled by default; an organisation owner/admin may disable it |
| `off` | Unavailable; neither organisation nor platform settings can enable it |
For example, `FEATURE_EVALS=adoption` exposes an org opt-in, while `FEATURE_EVALS=off` disables Evals for every organisation. Overrides take effect after restarting the affected processes. Boolean values such as `true` and `false` and the former feature grade `preview` are no longer supported; invalid values disable the feature. When migrating a deployment that offered org opt-in with `true`, use `adoption` to preserve that behavior; use `off` for former `false` values. `prod` intentionally enables the feature by default for every organisation. Configure the same grades on the API, workers, and UI where they perform registration or ingress gating.
## API\_PREFIX
Path prefix for REST API routes.
| Property | Value |
| ------------ | ------ |
| **Required** | No |
| **Default** | `/api` |
**Example:**
```bash
# Routes at /api/v1/agents
API_PREFIX=/api
```
**Notes:**
* `/health`, `/api-doc/openapi.json`, `/mcp`, `/.well-known/*`, `/oauth/*`, and `/cli/login-success` stay at the server root
* `/mcp` is always mounted and authenticated; there is no `FEATURE_MCP_ENDPOINT` deployment variable or organization feature toggle
* REST API routes including auth (`/v1/auth/*`) are mounted under this prefix
* OAuth callback URLs use `AUTH_BASE_URL`, which defaults to `PUBLIC_APP_URL` plus `API_PREFIX` when unset
* Override only if you need a non-`/api` REST prefix behind a reverse proxy or gateway
## PUBLIC\_APP\_URL
Public browser origin for the Everruns app. In single-origin deployments, set this once and the server derives `FRONTEND_URL` and `AUTH_BASE_URL` from it.
| Property | Value |
| ------------ | ----------------------- |
| **Required** | No |
| **Default** | `http://localhost:9300` |
**Example:**
```bash
PUBLIC_APP_URL=https://everruns.example.com
```
**Notes:**
* `FRONTEND_URL` defaults to `PUBLIC_APP_URL`
* `AUTH_BASE_URL` defaults to `PUBLIC_APP_URL` plus `API_PREFIX` (for example, `https://everruns.example.com/api`)
* Set `FRONTEND_URL` only when browser redirects must land on a different origin
* Set `AUTH_BASE_URL` only when OAuth callbacks use a different public API base
* In the UI runtime, it is the fallback API base for server-rendered pages; see [`UI_SERVER_API_URL`](#ui_server_api_url)
## AUTH\_LOGIN\_ORIGIN
Trusted browser origin that hosts the login page. Set this when an OSS-UI-based app delegates `/login` to a central identity origin.
| Property | Value |
| ------------ | --------------------------------------- |
| **Required** | No |
| **Default** | Not set (same-origin relative `/login`) |
**Example:**
```bash
AUTH_LOGIN_ORIGIN=https://id.example.com
```
**Notes:**
* Supply only an HTTP(S) origin, with no credentials, path, query, or fragment
* Set the same value in the server and UI runtime environments
* The value is trusted deployment configuration; request/query input cannot override it
* `return_to` remains a relative path and is still sanitized against open redirects
* Configured absolute login redirects use full-page navigation
## UI\_SERVER\_API\_URL
API base URL the UI server calls when it pre-renders a page (for example the agent, harness, session, and skill lists). Set it on the **UI** service. It may be an internal address that only the UI can reach.
| Property | Value |
| ------------ | ---------------------------------------------------------------------------------- |
| **Required** | No |
| **Default** | `PUBLIC_APP_URL` (as set in the UI runtime) plus `/api`; unset when neither is set |
**Example:**
```bash
UI_SERVER_API_URL=http://server:9000/api
```
**Notes:**
* Include the API prefix (`/api` by default); supply an HTTP(S) URL with no credentials, query, or fragment
* The UI forwards the signed-in user’s cookies on these requests, so the destination comes only from deployment configuration; the request’s `Host` header is never used
* Redirects are not followed; a redirect response counts as a failed prefetch
* If neither this nor `PUBLIC_APP_URL` is set in the UI runtime, or the value is invalid, the UI skips server prefetch and the browser loads the data after the page renders
## CORS\_ALLOWED\_ORIGINS
Comma-separated list of allowed origins for cross-origin requests. Only needed when the UI is served from a different domain than the API.
| Property | Value |
| ------------ | ----------------------- |
| **Required** | No |
| **Default** | Not set (CORS disabled) |
**Example:**
```bash
# Allow requests from a different frontend origin
CORS_ALLOWED_ORIGINS=https://app.example.com
# Multiple origins
CORS_ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com
```
**Notes:**
* Not needed for local development (Caddy reverse proxy keeps UI and backend on one origin)
* Not needed in production if using a reverse proxy on the same domain
* If set, credentials are allowed (`Access-Control-Allow-Credentials: true`)
* Wildcard (`*`) is not supported when using credentials
* Browsers may send `X-Org-Id`, `Idempotency-Key`, `Everruns-Change-Reason` and `Everruns-Context-Revision` in addition to the standard headers, and can read the `Idempotent-Replayed` response header
## HTTP\_ADDR
Bind address for the server HTTP API.
| Property | Value |
| ------------ | -------------- |
| **Required** | No |
| **Default** | `0.0.0.0:9000` |
**Example:**
```bash
HTTP_ADDR=0.0.0.0:9000
```
**Notes:**
* `ADDR` is supported as a legacy alias
* Container images already default to `0.0.0.0:9000`; most deployments do not need to set this
## VALKEY\_URL
Connection URL for Valkey (Redis-compatible) used for distributed rate limiting across control-plane instances.
| Property | Value |
| ------------ | --------------------------------------------------- |
| **Required** | No |
| **Default** | Not set (uses per-instance in-memory rate limiting) |
**Example:**
```bash
# Local Valkey
VALKEY_URL=redis://localhost:6379
# With authentication
VALKEY_URL=redis://user:password@valkey.example.com:6379
# TLS (managed cloud service)
VALKEY_URL=rediss://user:password@valkey.example.com:6380
```
**Notes:**
* When not set, rate limiting falls back to in-memory governor (per-instance, no coordination)
* With N instances behind a load balancer, per-instance rate limiting allows N× the intended budget per IP, set `VALKEY_URL` for coordinated limits
* Accepts `redis://`, `rediss://` (TLS), `valkey://`, `valkeys://` (TLS) schemes
* Fail-open: if Valkey is unreachable, requests are allowed (availability over strictness)
* Only used by control-plane (server); workers don’t need this variable
* Uses sliding-window counters via Lua scripts for atomic rate limit checks
## DATABASE\_URL
PostgreSQL connection URL for the control plane. Workers do not connect to the database; they reach it through the API over gRPC.
| Property | Value |
| ------------ | ------------------------------------------------------------------------- |
| **Required** | Yes, unless `DEV_MODE=true` |
| **Default** | Not set (startup fails with `DATABASE_URL environment variable required`) |
**Example:**
```bash
DATABASE_URL=postgres://everruns:secret@db.example.com:5432/everruns?sslmode=require
```
**Notes:**
* Startup warns when the URL has no `sslmode` or uses `sslmode=disable`.
* When this points at a pooler, set [`DATABASE_UNPOOLED_URL`](#database_unpooled_url) for listeners.
## DATABASE\_UNPOOLED\_URL
Direct PostgreSQL connection URL used only for session-scoped `LISTEN/NOTIFY` listeners.
| Property | Value |
| ------------ | --------------------------------------------------------------------- |
| **Required** | No |
| **Default** | Not set (listeners reuse `DATABASE_URL` if it is a direct connection) |
**Example:**
```bash
# Query traffic through a pooler, listeners through a direct endpoint
DATABASE_URL=postgres://app:secret@ep-foo-pooler.us-east-1.aws.neon.tech/everruns?sslmode=require
DATABASE_UNPOOLED_URL=postgres://app:secret@ep-foo.us-east-1.aws.neon.tech/everruns?sslmode=require
```
**Notes:**
* Use this when `DATABASE_URL` points at Neon `-pooler`, PgBouncer, or another proxy that does not preserve session-scoped `LISTEN/NOTIFY` semantics.
* Listener paths include PostgreSQL-backed event wakeups, notification SSE, and PG task notification fallback when NATS is unavailable.
* If `DATABASE_URL` or `DATABASE_UNPOOLED_URL` appears to point at a pooled/proxied endpoint, startup now fails fast with guidance to set a direct listener URL.
* Ordinary query traffic still uses `DATABASE_URL`.
## Object Storage (S3-compatible blob backend)
Optional backend that offloads workspace-file and image *content bytes* to an S3-compatible object store while keeping all metadata in PostgreSQL. Everruns remains the proxy for every read/write, no presigned URLs are handed to clients or workers.
| Variable | Required | Default | Description |
| ------------------------------------- | --------- | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `STORAGE_BLOB_BACKEND` | No | `db` | `db` keeps bytes inline in PostgreSQL (current behavior); `s3` offloads to object storage. |
| `STORAGE_S3_BUCKET` | When `s3` | , | Target bucket name. |
| `STORAGE_S3_REGION` | No | , | Bucket region / region label. |
| `STORAGE_S3_ENDPOINT` | No | , | Custom endpoint for S3-compatible stores (SeaweedFS, R2). Unset for AWS S3. |
| `STORAGE_S3_ACCESS_KEY_ID` | No | , | Static access key. Omit to use the AWS credential chain (IAM role/instance). |
| `STORAGE_S3_SECRET_ACCESS_KEY` | No | , | Static secret key. |
| `STORAGE_S3_PREFIX` | No | (empty) | Key prefix isolating multiple deployments within one bucket. |
| `STORAGE_S3_ALLOW_HTTP` | No | `false` | Allow plaintext HTTP (local/dev only, e.g. SeaweedFS over HTTP). |
| `STORAGE_S3_FORCE_PATH_STYLE` | No | `true` | Use path-style requests (required by SeaweedFS; harmless on AWS S3). |
| `STORAGE_BLOB_GC_INTERVAL_SECONDS` | No | `21600` (6h) | Interval between blob GC sweeps that reclaim orphaned objects. `0` disables GC. Only effective with the `s3` backend (inline `db` storage has no orphans). |
| `STORAGE_BLOB_GC_GRACE_SECONDS` | No | `86400` (24h) | Safety grace period; orphaned objects younger than this are never deleted (avoids racing in-flight creates). |
| `STORAGE_BLOB_GC_MAX_DELETES_PER_RUN` | No | `10000` | Per-sweep deletion cap to bound work; remaining orphans are reclaimed next sweep. |
| `STORAGE_BLOB_GC_MAX_LIST_PER_RUN` | No | `100000` | Per-sweep cap on objects listed per prefix, bounding GC memory; larger buckets are reconciled across sweeps in key-order windows. |
**Example (local SeaweedFS via `just seaweedfs`):**
```bash
STORAGE_BLOB_BACKEND=s3
STORAGE_S3_BUCKET=everruns-dev
STORAGE_S3_ENDPOINT=http://127.0.0.1:8333
STORAGE_S3_REGION=us-east-1
STORAGE_S3_ACCESS_KEY_ID=everruns
STORAGE_S3_SECRET_ACCESS_KEY=everruns-secret
STORAGE_S3_ALLOW_HTTP=true
```
Any S3-compatible store works (AWS S3, SeaweedFS, R2); only the endpoint and credentials differ.
**Notes:**
* The backend is selected per process at startup; a deployment runs entirely on `db` or `s3`. Enabling `s3` offloads newly written content; pre-existing inline content is still served transparently.
* Tenant isolation is by object key (`workspaces/{workspace_id}/…`, `images/org-{org_id}/…`) plus the existing org/workspace authorization layer.
## NATS\_URL
Connection URL for NATS with JetStream, used for push-based event delivery and task notifications.
| Property | Value |
| ------------ | ------------------------------------------------------------------------------------------- |
| **Required** | No |
| **Default** | Not set (uses PG NOTIFY for task notifications, in-memory broadcast for SSE event delivery) |
**Example:**
```bash
# Local NATS
NATS_URL=nats://localhost:4222
# Cluster
NATS_URL=nats://nats1:4222,nats://nats2:4222,nats://nats3:4222
# Server with `authorization { users: [...] }`
NATS_URL=nats://control:s3cret@nats:4222
```
**Notes:**
* When not set, the system behaves exactly as before, all events persist to PG, SSE polls PG, task notifications use PG NOTIFY. Zero behavioral change.
* When set, enables two features:
* **Ephemeral event delivery**: delta events (`output.message.delta`, `reason.thinking.delta`, `tool.output.delta`, `llm.generation`) skip PostgreSQL and flow only through NATS JetStream. SSE streams subscribe to NATS instead of polling PG.
* **Task notifications**: `task.available.{activity_type}` subjects replace PG NOTIFY for push-based worker notification. Lower latency (\~1ms vs \~30ms), supports multi-instance deployments.
* When NATS event delivery is active, the server skips the legacy PostgreSQL event listener used only for SSE wakeups.
* NATS JetStream must be enabled on the server (`--jetstream` flag)
* Credentials embedded in the URL (`nats://user:password@host`) are sent as user/password auth. Reserved characters in the password (`/`, `@`, `:`, `%`) are accepted as-is, so a generated secret can be pasted unmodified; percent-encoded passwords are decoded and also work
* Fail-graceful: if NATS connection fails at startup, falls back to PG NOTIFY + in-memory delivery with a warning that includes the connection error
* Only used by control-plane (server); workers communicate via gRPC and don’t need NATS access
* Default port: 4222 (or `22` with `PORT_PREFIX`, for example `27122` for prefix `271`)
* `just start-all` automatically starts NATS and exports `NATS_URL` if `nats-server` is installed
## LLM Provider API Keys
LLM provider API keys (OpenAI, Anthropic, Gemini) are primarily stored encrypted in the database and managed via the Models > Providers UI.
| Property | Value |
| ----------------------- | -------------------------------------------- |
| **Storage** | Database (encrypted with AES-256-GCM) |
| **Configuration** | Models > Providers UI or `/v1/providers` API |
| **Supported Providers** | OpenAI, Anthropic, Google Gemini |
**Required for encryption:**
The `SECRETS_ENCRYPTION_KEY` environment variable must be set for the control-plane API to encrypt/decrypt API keys. Workers receive decrypted API keys via gRPC and do not need this variable.
```bash
# Generate a new key
python3 -c "import os, base64; print('kek-v1:' + base64.b64encode(os.urandom(32)).decode())"
# Set in environment (control-plane only)
SECRETS_ENCRYPTION_KEY=kek-v1:your-generated-key-here
```
### Default API Keys (Development Convenience)
For development, you can set default API keys via environment variables on the **control-plane only**. These are used as fallbacks when providers don’t have keys configured in the database.
| Variable | Description |
| --------------------------- | --------------------------------------------- |
| `DEFAULT_OPENAI_API_KEY` | Fallback API key for OpenAI providers |
| `DEFAULT_ANTHROPIC_API_KEY` | Fallback API key for Anthropic providers |
| `DEFAULT_GEMINI_API_KEY` | Fallback API key for Google Gemini providers |
| `DEFAULT_META_API_KEY` | Fallback API key for Meta Model API providers |
**Example:**
```bash
# Set in .env or environment (control-plane only)
DEFAULT_OPENAI_API_KEY=sk-...
DEFAULT_ANTHROPIC_API_KEY=sk-ant-...
DEFAULT_GEMINI_API_KEY=AIza...
DEFAULT_META_API_KEY=...
```
**Notes:**
* These variables are only used by the control-plane, not workers
* Workers receive API keys via gRPC from the control-plane
* Database-stored keys always take priority over environment variables
* These are intended for development convenience, not production use
* The `just start-all` command automatically sets these from `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, and `GEMINI_API_KEY` if present
* If no API key is configured for a provider, LLM calls will fail and users will see an error message in the chat: “I encountered an error while processing your request. Please try again later.”
## System Model Keys
Two deployment-owned models sit outside the provider system above. Neither is selectable by an agent, neither is stored in the database, and neither is reachable from session or agent configuration — they are host services the platform uses for its own internal work.
| Variable | Powers | Unset means |
| ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UTILITY_OPENAI_API_KEY` | Agent Analyze/Health checks, and guardrail checks with `engine: "utility_llm"` (the default), called directly against OpenAI | Those checks are skipped; Analyze and Health are unavailable |
| `UTILITY_OPENROUTER_API_KEY` | The same work, routed through OpenRouter instead. Setting it selects OpenRouter; it wins when both keys are set, and the startup log says so | The utility LLM falls back to `UTILITY_OPENAI_API_KEY` |
| `UTILITY_LLM_MODEL` | The model the utility LLM calls on whichever backend was selected | Defaults to `gpt-6-luna` on OpenAI, `openai/gpt-6-luna` on OpenRouter |
| `UTILITY_TYPESAFE_API_KEY` | Guardrail checks with `engine: "jev"`, answered by TypeSafe | Those checks are skipped with a warning and the turn proceeds, unless `UTILITY_DECISION_DRIVER` picks another driver |
| `UTILITY_AZURE_AI_API_KEY`, `UTILITY_AZURE_AI_ENDPOINT` | Microsoft-Decision-1 on Microsoft Foundry for `jev` checks: the resource key and its resource or project endpoint (`https://.services.ai.azure.com`). Used only when `UTILITY_DECISION_DRIVER=mai` | The `mai` driver is unavailable. Setting only one of the two stops startup |
| `UTILITY_DECISION_DRIVER` | Which decision driver answers `jev` checks and the Slack respond-or-not check: `typesafe`, `openai` (OpenAI’s Decisions API, using `UTILITY_OPENAI_API_KEY`), `openrouter` (Jev through `UTILITY_OPENROUTER_API_KEY`), `mai` (Microsoft-Decision-1 on Foundry, using `UTILITY_AZURE_AI_*`), or `llm` to answer them with the utility LLM when you have no TypeSafe key. `llm` answers are labels, not calibrated probabilities | `typesafe` when its key is set, otherwise disabled. A driver that is not configured stops startup. Replaces `DECISIONS_DRIVER`, which now stops startup |
| `UTILITY_DECISION_MODEL` | The model the chosen decision driver is asked for. For `mai` it is the Foundry deployment name | The driver’s own default (`Microsoft-Decision-1` for `mai`). Not allowed with `UTILITY_DECISION_DRIVER=llm`, which uses `UTILITY_LLM_MODEL`. Replaces `DECISIONS_MODEL` |
The keys are read from the process environment at startup. Missing keys **fail open**: a guardrail whose engine is not configured never blocks, so a missing key weakens policy rather than wedging traffic. Check the startup logs if a configured guardrail appears to do nothing.
There is no separate provider variable: the utility LLM’s backend is whichever key you supply. An OpenRouter model id is namespaced by its upstream provider, so override `UTILITY_LLM_MODEL` with an id that backend accepts (`anthropic/claude-sonnet-5.5`, not `claude-sonnet-5.5`).
```bash
# Control-plane and workers both read these.
UTILITY_OPENAI_API_KEY=sk-...
# ...or route the utility LLM through OpenRouter instead:
UTILITY_OPENROUTER_API_KEY=sk-or-...
UTILITY_LLM_MODEL=openai/gpt-6-luna
UTILITY_TYPESAFE_API_KEY=ts-...
# ...or answer jev checks with OpenAI's Decisions API:
UTILITY_DECISION_DRIVER=openai
```
Agents can also be given the TypeSafe capability directly, which is a **separate** credential: a per-user connection configured in Settings > Connections, never this deployment key. See [TypeSafe](https://everruns.com/docs/integrations/typesafe/) and [Guardrails](https://everruns.com/docs/capabilities/guardrails/).
## System Email Delivery
System email delivery is an internal service used by product and operational flows. It is not an agent capability, public API, or UI setting.
| Variable | Required | Default | Description |
| --------------------- | ------------------------------------- | ------------------------ | ---------------------------------------------------------------- |
| `EMAIL_PROVIDER` | Yes, when sending email in production | unset / disabled | Email provider. Supported values: `disabled`, `resend` |
| `RESEND_API_KEY` | Yes, when `EMAIL_PROVIDER=resend` | unset | Resend API key |
| `RESEND_API_BASE_URL` | No | `https://api.resend.com` | Resend API base URL override for tests or controlled deployments |
**Example:**
```bash
EMAIL_PROVIDER=resend
RESEND_API_KEY=re_...
```
**Notes:**
* Set these on the control-plane process that performs system email sends.
* The sender is fixed in code as `Everruns `.
* The Resend account must have `everruns.com` verified and enabled for sending.
## UI API Proxy Architecture
The UI makes all REST API requests (including SSE) to `/api/*` paths. The backend serves those routes under `/api` directly. Root-level backend routes like `/oauth/*`, `/mcp`, and `/.well-known/*` bypass the UI and are proxied straight to the backend.
**Local Development:**
* Caddy on `:9300` routes `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to backend at `:9301`
* Example: `/api/v1/agents` → `http://localhost:9301/api/v1/agents`
* Example: `/oauth/authorize?...` → `http://localhost:9301/oauth/authorize?...`
* Example: `/mcp` → `http://localhost:9301/mcp`
* Example: `/.well-known/oauth-authorization-server` → `http://localhost:9301/.well-known/oauth-authorization-server`
* SSE streaming works via `flush_interval -1` in Caddy config
* No CORS needed (same-origin through Caddy)
**Production:**
* Configure your reverse proxy (nginx, Caddy, etc.) to route `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to the API server
* Disable response buffering for SSE endpoints
* [Example Caddy configuration](https://github.com/everruns/everruns/blob/main/local/Caddyfile)
## Other Server Variables
| Variable | Default | Description |
| ------------------------------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AUTH_MODE` and `AUTH_*` | Depends on `DEPLOYMENT_GRADE` | Authentication mode and its settings (`AUTH_BASE_URL`, `AUTH_JWT_SECRET`, `AUTH_LOGIN_ORIGIN`, and others). See [Authentication](https://everruns.com/docs/sre/runbooks/authentication/). |
| `RATE_LIMIT_API_REQUESTS_PER_MINUTE` | `1200` | API request rate limit per minute. Uses Valkey when `VALKEY_URL` is set, otherwise a per-instance in-memory limiter. |
| `TRUSTED_PROXY_HOPS` | `1` | Number of trusted reverse proxies in front of the server. The client IP is taken from `X-Forwarded-For` counting this many entries from the right. Raise it if a CDN or load balancer sits in front of your reverse proxy. |
| `EVENT_RETENTION_DAYS` | `0` (disabled) | Archive session events older than this many days. A background task runs hourly. |
| `DATABASE_POOL_MAX` | `50` | Maximum PostgreSQL connections in the request pool. |
| `DATABASE_POOL_MIN` | `5` | Minimum PostgreSQL connections kept open. |
| `SECRETS_ENCRYPTION_KEY_PREVIOUS` | Unset | Previous encryption key, kept during key rotation so existing values still decrypt. See [Encryption key rotation](https://everruns.com/docs/sre/runbooks/encryption-key-rotation/). |
| `EXPECTED_INSTANCES` | `1` | Number of control-plane instances behind the load balancer. Divides the global and per-org SSE connection limits across instances. Per-session limits are not divided. |
## SSE Streaming Configuration
| Variable | Default | Description |
| ----------------------------- | ------- | ------------------------------------------------------------------ |
| `SSE_REALTIME_CYCLE_SECS` | `300` | Connection cycle interval for session event streams (seconds) |
| `SSE_MONITORING_CYCLE_SECS` | `600` | Connection cycle interval for durable monitoring streams (seconds) |
| `SSE_HEARTBEAT_INTERVAL_SECS` | `30` | Interval between heartbeat comments on all SSE streams (seconds) |
| `SSE_GLOBAL_MAX` | `10000` | Maximum total SSE connections across all users |
| `SSE_PER_SESSION_MAX` | `12` | Maximum SSE connections per session |
| `SSE_PER_ORG_MAX` | `1000` | Maximum SSE connections per organization |
**Notes:**
* Heartbeat comments (`: heartbeat\n\n`) are sent on all SSE streams to detect stale connections
* The heartbeat interval must be less than the SDK read timeout (default: 45s) with safety margin
* Connection cycling prevents stale connections through proxies and load balancers
* When running behind HTTP/1.1 proxies, increase `SSE_REALTIME_CYCLE_SECS` to reduce reconnection frequency
## Worker gRPC Configuration
### SERVER\_GRPC\_ADDRESS
Address of the server gRPC endpoint for worker communication.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (worker only) |
| **Default** | `127.0.0.1:9001` |
**Example:**
```bash
SERVER_GRPC_ADDRESS=127.0.0.1:9001
```
**Notes:**
* Workers communicate with the server via gRPC for all database operations
* `WORKER_GRPC_ADDRESS` is supported as a legacy alias
* The server exposes both HTTP (default `9000`) and gRPC (default `9001`) interfaces
* Workers are stateless and do not connect directly to the database
### WORKER\_GRPC\_AUTH\_TOKEN
Bearer token for authenticating worker gRPC connections to the control-plane.
| Property | Value |
| ------------ | ------------------------------- |
| **Required** | Yes (production); No (dev mode) |
| **Default** | Unset (auth disabled) |
**Example:**
```bash
WORKER_GRPC_AUTH_TOKEN=your-secret-token
```
**Notes:**
* Must be set on both the server and all workers (same value)
* When unset, gRPC auth is disabled (acceptable for local development only)
* Server panics on startup if unset when not in dev mode
### SERVER\_GRPC\_BIND\_ADDR
Bind address for the server-side gRPC listener.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (server only) |
| **Default** | `0.0.0.0:9001` |
**Example:**
```bash
SERVER_GRPC_BIND_ADDR=0.0.0.0:9001
```
**Notes:**
* `WORKER_GRPC_ADDR` is supported as a legacy alias
### WORKER\_GRPC\_CONNECT\_TIMEOUT
Timeout in seconds for worker initial connection to control-plane gRPC.
| Property | Value |
| ------------ | ---------------- |
| **Required** | No (worker only) |
| **Default** | `30` |
**Example:**
```bash
WORKER_GRPC_CONNECT_TIMEOUT=60
```
### WORKER\_GRPC\_TLS\_CERT
Path to PEM-encoded certificate file. On the server, this is the gRPC server certificate. On the worker, this is the client certificate presented during mTLS handshake.
| Property | Value |
| ------------ | ---------------------- |
| **Required** | No |
| **Default** | Not set (TLS disabled) |
**Example:**
```bash
WORKER_GRPC_TLS_CERT=/etc/everruns/grpc-cert.pem
```
**Notes:**
* Must be set together with `WORKER_GRPC_TLS_KEY`
* Server: enables TLS on the gRPC listener when both cert and key are set
* Worker: presents client certificate to the server when both cert and key are set (requires `WORKER_GRPC_TLS_CA_CERT`)
### WORKER\_GRPC\_TLS\_KEY
Path to PEM-encoded private key file corresponding to `WORKER_GRPC_TLS_CERT`.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
WORKER_GRPC_TLS_KEY=/etc/everruns/grpc-key.pem
```
### WORKER\_GRPC\_TLS\_CA\_CERT
Path to PEM-encoded CA certificate bundle for verifying the remote peer.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
WORKER_GRPC_TLS_CA_CERT=/etc/everruns/grpc-ca.pem
```
**Notes:**
* Server: when set, requires workers to present valid client certificates signed by this CA (mutual TLS)
* Worker: when set, verifies the server’s certificate against this CA and switches to `https://` transport
* For full mTLS, set on both server and worker alongside their respective cert/key pairs
### WORKER\_GRPC\_TLS\_DOMAIN
Override the expected server domain name for TLS certificate verification (worker only).
| Property | Value |
| ------------ | ------------------------------------------- |
| **Required** | No |
| **Default** | Derived from `SERVER_GRPC_ADDRESS` hostname |
**Example:**
```bash
WORKER_GRPC_TLS_DOMAIN=control-plane.internal
```
**Notes:**
* Useful when the server certificate CN/SAN differs from the connection hostname (e.g., connecting via IP but cert has a DNS name)
## OpenTelemetry Configuration
Everruns supports distributed tracing via OpenTelemetry with OTLP export. Agent traces follow the OpenTelemetry [Gen-AI agent and inference conventions](https://github.com/open-telemetry/semantic-conventions-genai/tree/main/docs/gen-ai) and, on the same spans, the [OpenInference conventions](https://arize-ai.github.io/openinference/spec/semantic_conventions.html) read by Arize Phoenix, so one OTLP endpoint serves both families of backends.
### OTEL\_EXPORTER\_OTLP\_ENDPOINT
OTLP endpoint for trace export (e.g., Grafana Tempo, Arize Phoenix, Datadog, or any OTLP-compatible backend).
| Property | Value |
| ------------ | -------------------------- |
| **Required** | No |
| **Default** | Not set (tracing disabled) |
**Example:**
```bash
# For a local OTLP collector or Phoenix
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# For production Tempo
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.monitoring:4318
```
**Notes:**
* When set, traces are exported via OTLP over HTTP/protobuf (use the backend’s HTTP port, typically 4318, not the gRPC port 4317)
* Point this at the base endpoint; the `/v1/traces` path is appended for you, and a full signal URL is used as given
* Connect to any OTLP-compatible backend for trace visualization
* See [OpenTelemetry](https://everruns.com/docs/observability/opentelemetry/) for the span model and attributes
* Without this variable, only console logging is enabled
### OTEL\_SERVICE\_NAME
Service name for traces.
| Property | Value |
| ------------ | --------------------------------------------------- |
| **Required** | No |
| **Default** | `everruns-server` (API), `everruns-worker` (Worker) |
**Example:**
```bash
OTEL_SERVICE_NAME=everruns-prod-api
```
### OTEL\_SERVICE\_VERSION
Service version for traces.
| Property | Value |
| ------------ | --------------------- |
| **Required** | No |
| **Default** | Cargo package version |
### OTEL\_ENVIRONMENT
Deployment environment label.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | Not set |
**Example:**
```bash
OTEL_ENVIRONMENT=production
```
### OTEL\_RECORD\_CONTENT
Enable recording of LLM input/output content in traces. **Warning:** May contain sensitive data.
| Property | Value |
| ------------ | ------- |
| **Required** | No |
| **Default** | `false` |
**Example:**
```bash
# Standard OTel env var (preferred)
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
# Legacy alias (also works)
OTEL_RECORD_CONTENT=true
```
**Notes:**
* When enabled, the chat span records `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.tool.definitions` (plus the OpenInference `input.value`, `output.value`, and flattened `llm.input_messages.*`); tool spans record `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` (plus `everruns.tool.executed_arguments` when a pre-tool hook rewrote the call); the turn root records the input message and final answer; the thinking span records the reasoning text
* Disabled by default for privacy and data size concerns
* Only enable in development or when debugging specific issues
### EVERRUNS\_TRACE\_CONVENTIONS
Which attribute vocabularies agent spans carry.
| Property | Value |
| ------------ | ---------------------- |
| **Required** | No |
| **Default** | `gen_ai,openinference` |
**Example:**
```bash
# Only the OpenTelemetry Gen-AI attributes (Tempo, Jaeger, Datadog, Langfuse)
EVERRUNS_TRACE_CONVENTIONS=gen_ai
# Only the OpenInference attributes (Arize Phoenix)
EVERRUNS_TRACE_CONVENTIONS=openinference
```
**Notes:**
* Span names, kinds, and hierarchy are the same under either vocabulary; only attributes differ
* Unknown values are ignored, and an empty selection falls back to both
## Local Development with OpenTelemetry
To visualize traces locally, point `OTEL_EXPORTER_OTLP_ENDPOINT` at any OTLP-compatible collector:
```bash
# Set OTLP endpoint for API and Worker
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Start services
just start-all
```
To see traces in Arize Phoenix, run Phoenix locally and point the same variable at its OTLP/HTTP port (`http://localhost:6006`); spans render as AGENT, LLM, and TOOL spans with token counts and, when content capture is on, the messages.
### Gen-AI Trace Structure
Traces follow the agentic execution lifecycle with 13 event types; every span starts and ends at the timestamp of the event it records:
```plaintext
invoke_agent {agent name} (root span, INTERNAL)
├── reason (LLM reasoning phase)
│ └── chat {model} (LLM API call, CLIENT)
│ └── thinking (extended thinking, if enabled)
├── act (tool execution phase)
│ ├── execute_tool {name}
│ └── execute_tool {name}
├── reason (iteration 2)
│ └── chat {model}
└── ...
```
### Gen-AI Trace Attributes
Spans carry the OpenTelemetry Gen-AI attributes and the OpenInference attributes side by side (see `EVERRUNS_TRACE_CONVENTIONS`). The most useful ones:
| Attribute | Span Types | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `gen_ai.operation.name` | invoke\_agent, chat, execute\_tool | `invoke_agent`, `chat`, or `execute_tool` |
| `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.description` | invoke\_agent | Agent the turn runs as |
| `gen_ai.conversation.id` / `session.id` | All | Session identifier |
| `gen_ai.provider.name` / `llm.provider` | chat | Provider (`openai`, `anthropic`, `gcp.gemini`, `aws.bedrock`, …) |
| `gen_ai.request.model`, `gen_ai.response.model` / `llm.model_name` | chat | Model name |
| `gen_ai.response.id` | chat | Provider response identifier |
| `gen_ai.response.finish_reasons` | chat | Why generation stopped (string array) |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` / `llm.token_count.*` | chat, invoke\_agent | Token usage (turn root carries the cumulative total) |
| `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens` | chat, invoke\_agent | Prompt-cache tokens |
| `gen_ai.request.temperature`, `gen_ai.request.max_tokens`, `gen_ai.request.reasoning.level`, `gen_ai.request.stream` / `llm.invocation_parameters` | chat | Request parameters |
| `gen_ai.response.time_to_first_chunk` | chat | Streaming latency in seconds |
| `gen_ai.conversation.compacted` | chat | Context was compacted before the call |
| `llm.cost.total` / `everruns.usage.cost_usd` | chat, invoke\_agent | Cost in USD when known |
| `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`, `gen_ai.tool.description` / `tool.name`, `tool.description` | execute\_tool | Tool identity |
| `openinference.span.kind` | All | `AGENT`, `LLM`, `TOOL`, or `CHAIN` |
| `error.type` | All | Low-cardinality error class on failure (error code, HTTP status, `timeout`, or `_OTHER`); the span status carries the message |
| `everruns.phase` | reason, act, thinking | Phase span marker |
| `everruns.turn.id`, `everruns.exec.id` | All | Everruns correlation ids |
| `everruns.turn.iterations`, `everruns.turn.tool_call_count`, `everruns.turn.llm_call_count` | invoke\_agent | Turn counters |
| `everruns.tool.status` | execute\_tool | `success`, `error`, `timeout`, or `cancelled` |
## Braintrust Integration
Everruns supports sending turn, reasoning, tool, and session lifecycle events to [Braintrust](https://www.braintrust.dev/) for observability, evaluation, and logging.
For setup instructions and configuration details, see the [Braintrust Integration Guide](https://everruns.com/docs/observability/braintrust/).
| Variable | Required | Default | Description |
| -------------------------------- | -------- | ------------------------------- | --------------------------------------------------------------- |
| `BRAINTRUST_ENABLED` | No | enabled when API key is present | Explicit Braintrust on/off switch |
| `BRAINTRUST_API_KEY` | Yes | - | API key from Braintrust settings |
| `BRAINTRUST_PROJECT_NAME` | No | `My Project` | Project name for organizing traces |
| `BRAINTRUST_PROJECT_ID` | No | - | Direct project UUID (skips name lookup) |
| `BRAINTRUST_API_URL` | No | `https://api.braintrust.dev` | API base URL |
| `BRAINTRUST_QUEUE_CAPACITY` | No | `1024` | Buffered event capacity before new exports are dropped |
| `BRAINTRUST_MAX_BATCH_SIZE` | No | `50` | Max events per Braintrust insert call |
| `BRAINTRUST_FLUSH_INTERVAL_MS` | No | `500` | Max delay before flushing a partial batch |
| `BRAINTRUST_REQUEST_TIMEOUT_MS` | No | `10000` | Per-request timeout for Braintrust insert calls |
| `BRAINTRUST_MAX_RETRIES` | No | `3` | Retries for `429`, `5xx`, and timeout/connect failures |
| `BRAINTRUST_RETRY_BASE_DELAY_MS` | No | `250` | Initial retry backoff |
| `BRAINTRUST_RETRY_MAX_DELAY_MS` | No | `5000` | Retry backoff cap |
| `BRAINTRUST_RECORD_CONTENT` | No | `false` | Export raw turn and LLM text content |
| `BRAINTRUST_RECORD_THINKING` | No | `none` | Extended thinking export mode: `none`, `summary`, `full` |
| `BRAINTRUST_TOOL_ARGS_MODE` | No | `redacted` | Tool argument export mode: `full`, `redacted`, `none` |
| `BRAINTRUST_TOOL_RESULTS_MODE` | No | `summary` | Tool result export mode: `full`, `summary`, `redacted`, `none` |
| `BRAINTRUST_DEBUG_PAYLOADS` | No | `false` | Print full outbound Braintrust payload JSON to local debug logs |