Operations: environment variables, admin container, and SRE runbooks
---
# Admin Container
> Admin container tools for checking database migration status, rotating encryption keys, and running diagnostics.
Source:
The admin container provides tools for key rotation, migration status checks, and other administrative tasks in production environments. > **Note**: Migrations are **auto-applied on server startup**. Use `migrate-info` to inspect migration status. The `migrate` command applies pending migrations and changes the database. ## Building
```bash
docker build --target admin -f docker/Dockerfile.unified -t everruns-admin .
```
## Commands | Command | Description | | -------------- | ----------------------------------------------------------------- | | `migrate` | Run pending database migrations, the same way server startup does | | `migrate-info` | Show migration status | | `reencrypt` | Re-encrypt secrets with new key | | `shell` | Interactive shell for debugging | | `help` | Show usage information | ## Usage ### Check Migration Status Use this before deployments to verify migration state:
```bash
docker run --rm \
-e DATABASE_URL="postgres://user:pass@host:5432/db" \
everruns-admin migrate-info
```
### Run Migrations Manually Migrations auto-apply on server startup. Use this only for: * Running migrations without starting the server * Debugging migration issues (with `--no-migrations` on server)
```bash
docker run --rm \
-e DATABASE_URL="postgres://user:pass@host:5432/db" \
everruns-admin migrate
```
### Re-encrypt Secrets (Dry Run)
```bash
docker run --rm \
-e DATABASE_URL="postgres://user:pass@host:5432/db" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --dry-run
```
### Re-encrypt Secrets (Execute)
```bash
docker run --rm \
-e DATABASE_URL="postgres://user:pass@host:5432/db" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --batch-size 50
```
## Environment Variables | Variable | Required | Description | | --------------------------------- | ------------- | ---------------------------- | | `DATABASE_URL` | Yes | PostgreSQL connection string | | `SECRETS_ENCRYPTION_KEY` | For reencrypt | Primary encryption key | | `SECRETS_ENCRYPTION_KEY_PREVIOUS` | For rotation | Previous encryption key | | `RUST_LOG` | No | Log level (default: info) | ## TLS/SSL Connections The admin container supports TLS connections to PostgreSQL. Use the `sslmode` parameter in your connection string:
```bash
DATABASE_URL="postgres://user:pass@host:5432/db?sslmode=require"
```
## Migration Troubleshooting ### Migration Fails on Startup If the server won’t start due to a migration error: 1. Check server logs for the specific SQL error 2. Fix the migration file 3. Rebuild and redeploy ### Bad Migration Deployed If a migration succeeded but caused issues, use **forward-fix**:
```bash
# Create a new migration that fixes the problem
just new-migration fix_bad_migration
# Edit the migration, then redeploy
```
### Emergency: Manual Database Fix For emergencies where you need to manually fix the database:
```bash
# Start server without auto-migrations
everruns-server --no-migrations
# Connect and fix manually
psql -h host -U everruns -d everruns
> -- Fix schema issues
> DELETE FROM _sqlx_migrations WHERE version = 006; -- If needed
# Restart server normally
everruns-server
```
## Production Deployment The admin container can be run as a one-off task in any container orchestration platform: * **Kubernetes**: Use a Job or run via `kubectl run` * **ECS**: Use `aws ecs run-task` with command override * **Docker Compose**: Use `docker compose run` * **Nomad**: Use a batch job
---
# Environment Variables
> Every Everruns environment variable: database connections, authentication, encryption, and development mode.
Source:
## Summary A production control plane (any `DEPLOYMENT_GRADE` other than `dev`, `DEV_MODE` unset) refuses to start without the first three variables. Everything else has a working default; each row links to the section that documents it. | Variable | Process | Required | Without it | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`DATABASE_URL`](#database_url) | API | Yes, unless `DEV_MODE=true` | Startup fails. Dev mode ignores it and starts its own temporary PostgreSQL. | | [`AUTH_MODE`](#other-server-variables) | API | Yes, unless the grade is `dev` | Startup fails: unset means `none`, which is allowed only at `dev` grade. | | [`WORKER_GRPC_AUTH_TOKEN`](#worker_grpc_auth_token) | API and workers (same value) | Yes, unless `DEV_MODE=true` | The API fails to start its gRPC server; workers cannot authenticate. | | [`SECRETS_ENCRYPTION_KEY`](#llm-provider-api-keys) | API | Recommended | The API starts, but cannot store provider keys or other encrypted values. Turns still run, in-process and on remote workers, without durable compaction checkpoints or native async tool journals. | | [`DEV_MODE`](#dev_mode), [`DEPLOYMENT_GRADE`](#deployment_grade) | API and workers | No | Production behavior (`prod` grade). | | [`PUBLIC_APP_URL`](#public_app_url), [`AUTH_LOGIN_ORIGIN`](#auth_login_origin), [`API_PREFIX`](#api_prefix), [`CORS_ALLOWED_ORIGINS`](#cors_allowed_origins), [`HTTP_ADDR`](#http_addr) | API | No | `http://localhost:9300`, same-origin login, `/api`, CORS off, `0.0.0.0:9000`. | | [`DATABASE_UNPOOLED_URL`](#database_unpooled_url), [`DATABASE_POOL_MAX`](#other-server-variables), [`DATABASE_POOL_MIN`](#other-server-variables) | API | No | Listeners reuse `DATABASE_URL`; pool of 5 to 50 connections. | | [`VALKEY_URL`](#valkey_url) | API | No | Rate limits are per instance. | | [`NATS_URL`](#nats_url) | API | With more than one control-plane instance | PostgreSQL `NOTIFY` and in-memory broadcast (one instance only). | | [`STORAGE_*`](#object-storage-s3-compatible-blob-backend) | API | No | File and image bytes stay in PostgreSQL. | | [`SERVER_GRPC_ADDRESS`](#server_grpc_address), [`SERVER_GRPC_BIND_ADDR`](#server_grpc_bind_addr), [`WORKER_GRPC_CONNECT_TIMEOUT`](#worker_grpc_connect_timeout), [`WORKER_GRPC_REQUEST_TIMEOUT_SECS`](#worker_grpc_request_timeout_secs), [`WORKER_ACTOR_STALL_TIMEOUT_SECS`](#worker_actor_stall_timeout_secs), [`WORKER_TOOL_DETACH_AFTER_SECS`](#worker_tool_detach_after_secs), [`WORKER_MAX_RUNNING_TURNS`](#worker_max_running_turns), [`WORKER_MEMORY_LIMIT_MIB`](#worker_memory_limit_mib), [`WORKER_MAX_RUNTIME_LAG_MS`](#worker_max_runtime_lag_ms), [`MAX_DETACHED_TOOL_CALLS`](#max_detached_tool_calls), [`WORKER_GRPC_TLS_*`](#worker_grpc_tls_cert) | API or workers | No | Plain gRPC on `127.0.0.1:9001` / `0.0.0.0:9001`; 120 s per worker call; a stalled session is let go after 5 minutes; turns admitted by memory and runtime lag; a tool call running past 5 minutes parks its turn. | | [`DEFAULT_*_API_KEY`](#default-api-keys-development-convenience) | API | No | Providers need keys configured in Models > Providers. | | [`UTILITY_*`](#system-model-keys) | API and workers | No | Analyze, Health, and model-backed guardrail checks are unavailable or skipped. | | [`SLACK_MANAGER_APP_CLIENT_ID`](#slack-manager-app), [`SLACK_MANAGER_APP_CLIENT_SECRET`](#slack-manager-app) | API | No | Organizations connect Slack workspaces by pasting a configuration token. | | [`EVERRUNS_EGRESS_*`](#outbound-egress) | API and workers | No | Tenant traffic may reach any public host, directly from the deployment’s address. | | [`EGRESS_PROXY_*`](#tenant-egress-proxy) | `everruns-egress-proxy` | Only when running the proxy | No proxy. | | [`KUBERNETES_SANDBOX_*`](#isolated-sandboxes) | API and workers | Only for Isolated sandboxes | The Isolated sandbox target is off. | | [`EVERRUNS_DP_TOKEN`, `EVERRUNS_DP_KEY`, `EVERRUNS_DEFAULT_DATA_PLANE_URL`, `EVERRUNS_DATA_PLANE`](#data-plane-v2) | API (the token and key also on shards, as `AGENTD_DP_*`) | For data plane v2 (the Compose, Helm and local default) | New organizations run on the workers (v1). | | [`EMAIL_PROVIDER`](#system-email-delivery), [`RESEND_*`](#system-email-delivery) | API | When sending email | Email delivery is disabled. | | [`RATE_LIMIT_API_REQUESTS_PER_MINUTE`](#other-server-variables), [`TRUSTED_PROXY_HOPS`](#other-server-variables), [`EXPECTED_INSTANCES`](#other-server-variables), [`EVENT_RETENTION_DAYS`](#other-server-variables), [`SSE_*`](#sse-streaming-configuration), [`SERVER_SHUTDOWN_*`](#server-shutdown) | API | No | 1200 requests per minute, one trusted proxy, one instance, no event archiving, default SSE limits, a 5 s linger and 20 s drain on shutdown. | | [`EVENT_GROUP_COMMIT`](#throughput-tuning), [`SESSION_ACTOR_STICKY_GRACE_MS`](#throughput-tuning) | API | No | Event writes share commits over 4 lanes, a session waits 300 ms for the worker that ran its last turn. | | [`MONITORING_DETAIL`](#monitoring-detail), [`MONITORING_DETAIL_URL`](#monitoring-detail) | API | No | Monitoring keeps its one-minute history in PostgreSQL, without 10-second shard detail. | | [`OTEL_*`](#opentelemetry-configuration), [`EVERRUNS_TRACE_CONVENTIONS`](#everruns_trace_conventions), [`BRAINTRUST_*`](#braintrust-integration) | API and workers | No | Tracing and Braintrust export are off. | ## DEV_MODE Enable development mode. No PostgreSQL to install or run: the server starts a temporary PostgreSQL of its own and deletes it on exit. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | `false` | **Example:**
```bash
# Start in dev mode (no database to set up)
DEV_MODE=true ./target/debug/everruns-server
# Or with 1
DEV_MODE=1 ./target/debug/everruns-server
```
**Notes:** * When enabled, the server downloads PostgreSQL 17 binaries on first use (cached under `~/.cache/everruns/postgres`, or `EVERRUNS_PG_CACHE`), starts a cluster on a free loopback port, and runs migrations into a fresh database. `DATABASE_URL` is ignored. * Data does not survive a restart. Use `just start-all` for a persistent local database. * Run dev mode as a non-root user. Root callers receive a startup error; use a non-root account or external PostgreSQL with `DEV_MODE=false`. * All data is lost when the server stops * gRPC server and worker communication are disabled * Stale task reclamation is disabled * Useful for quick local development and testing * Not suitable for production or multi-instance deployments **Limitations in dev mode:** * No persistence (data is lost on restart) * No worker support (all execution happens in-process) * No distributed tracing of worker activities * Single-instance only ## DEPLOYMENT_GRADE Deployment environment grade. Controls which features and capabilities are available. | Property | Value | | ------------ | ------------------------------------ | | **Required** | No | | **Default** | `prod` (or `dev` if `DEV_MODE=true`) | **Valid values:** | Grade | Description | | --------- | ----------------------------------------------- | | `dev` | Development - all experimental features enabled | | `poc` | Proof of concept / demo environment | | `preview` | Preview/staging environment | | `prod` | Production - only stable features | **Example:**
```bash
# Run in development mode with experimental features
DEPLOYMENT_GRADE=dev ./target/debug/everruns-server
# Production mode (default)
DEPLOYMENT_GRADE=prod ./target/debug/everruns-server
```
**Notes:** * If not set, falls back to `DEV_MODE`: if `DEV_MODE=true`, uses `dev`; otherwise uses `prod` * Experimental capabilities (e.g., Docker Container) are only available in `dev` grade * Experimental seed agents (e.g., Python Coder) are only created in `dev` grade * Use `dev` for local development and testing experimental features * Use `prod` for production deployments ## Feature rollout grades Each `FEATURE_` variable overrides that feature’s rollout grade. Feature grades are separate from `DEPLOYMENT_GRADE`, which describes the running deployment. | Value | Behaviour | | ---------- | ------------------------------------------------------------------------- | | `dev` | Available only on a local dev deployment; enabled there by default | | `internal` | Disabled until a platform operator enables it for a specific organisation | | `adoption` | Disabled by default; an organisation owner/admin may enable it | | `prod` | Enabled by default; an organisation owner/admin may disable it | | `off` | Unavailable; neither organisation nor platform settings can enable it | For example, `FEATURE_EVALS=adoption` exposes an org opt-in, while `FEATURE_EVALS=off` disables Evals for every organisation. Overrides take effect after restarting the affected processes. Boolean values such as `true` and `false` and the former feature grade `preview` are no longer supported; invalid values disable the feature. When migrating a deployment that offered org opt-in with `true`, use `adoption` to preserve that behavior; use `off` for former `false` values. `prod` intentionally enables the feature by default for every organisation. Configure the same grades on the API, workers, and UI where they perform registration or ingress gating. ## API_PREFIX Path prefix for REST API routes. | Property | Value | | ------------ | ------ | | **Required** | No | | **Default** | `/api` | **Example:**
```bash
# Routes at /api/v1/agents
API_PREFIX=/api
```
**Notes:** * `/health`, `/api-doc/openapi.json`, `/mcp`, `/.well-known/*`, `/oauth/*`, and `/cli/login-success` stay at the server root * `/mcp` is always mounted and authenticated; there is no `FEATURE_MCP_ENDPOINT` deployment variable or organization feature toggle * REST API routes including auth (`/v1/auth/*`) are mounted under this prefix * OAuth callback URLs use `AUTH_BASE_URL`, which defaults to `PUBLIC_APP_URL` plus `API_PREFIX` when unset * Override only if you need a non-`/api` REST prefix behind a reverse proxy or gateway ## PUBLIC_APP_URL Public browser origin for the Everruns app. In single-origin deployments, set this once and the server derives `FRONTEND_URL` and `AUTH_BASE_URL` from it. | Property | Value | | ------------ | ----------------------- | | **Required** | No | | **Default** | `http://localhost:9300` | **Example:**
```bash
PUBLIC_APP_URL=https://everruns.example.com
```
**Notes:** * `FRONTEND_URL` defaults to `PUBLIC_APP_URL` * `AUTH_BASE_URL` defaults to `PUBLIC_APP_URL` plus `API_PREFIX` (for example, `https://everruns.example.com/api`) * Set `FRONTEND_URL` only when browser redirects must land on a different origin * Set `AUTH_BASE_URL` only when OAuth callbacks use a different public API base * In the UI runtime, it is the fallback API base for server-rendered pages; see [`UI_SERVER_API_URL`](#ui_server_api_url) ## AUTH_LOGIN_ORIGIN Trusted browser origin that hosts the login page. Set this when an OSS-UI-based app delegates `/login` to a central identity origin. | Property | Value | | ------------ | --------------------------------------- | | **Required** | No | | **Default** | Not set (same-origin relative `/login`) | **Example:**
```bash
AUTH_LOGIN_ORIGIN=https://id.example.com
```
**Notes:** * Supply only an HTTP(S) origin, with no credentials, path, query, or fragment * Set the same value in the server and UI runtime environments * The value is trusted deployment configuration; request/query input cannot override it * `return_to` remains a relative path and is still sanitized against open redirects * Configured absolute login redirects use full-page navigation ## UI_SERVER_API_URL API base URL the UI server calls when it pre-renders a page (for example the agent, harness, session, and skill lists). Set it on the **UI** service. It may be an internal address that only the UI can reach. | Property | Value | | ------------ | ---------------------------------------------------------------------------------- | | **Required** | No | | **Default** | `PUBLIC_APP_URL` (as set in the UI runtime) plus `/api`; unset when neither is set | **Example:**
```bash
UI_SERVER_API_URL=http://server:9000/api
```
**Notes:** * Include the API prefix (`/api` by default); supply an HTTP(S) URL with no credentials, query, or fragment * The UI forwards the signed-in user’s cookies on these requests, so the destination comes only from deployment configuration; the request’s `Host` header is never used * Redirects are not followed; a redirect response counts as a failed prefetch * If neither this nor `PUBLIC_APP_URL` is set in the UI runtime, or the value is invalid, the UI skips server prefetch and the browser loads the data after the page renders ## CORS_ALLOWED_ORIGINS Comma-separated list of allowed origins for cross-origin requests. Only needed when the UI is served from a different domain than the API. | Property | Value | | ------------ | ----------------------- | | **Required** | No | | **Default** | Not set (CORS disabled) | **Example:**
```bash
# Allow requests from a different frontend origin
CORS_ALLOWED_ORIGINS=https://app.example.com
# Multiple origins
CORS_ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com
```
**Notes:** * Not needed for local development (Caddy reverse proxy keeps UI and backend on one origin) * Not needed in production if using a reverse proxy on the same domain * If set, credentials are allowed (`Access-Control-Allow-Credentials: true`) * Wildcard (`*`) is not supported when using credentials * Browsers may send `X-Org-Id`, `Idempotency-Key`, `Everruns-Change-Reason` and `Everruns-Context-Revision` in addition to the standard headers, and can read the `Idempotent-Replayed` response header ## HTTP_ADDR Bind address for the server HTTP API. | Property | Value | | ------------ | -------------- | | **Required** | No | | **Default** | `0.0.0.0:9000` | **Example:**
```bash
HTTP_ADDR=0.0.0.0:9000
```
**Notes:** * `ADDR` is supported as a legacy alias * Container images already default to `0.0.0.0:9000`; most deployments do not need to set this ## Server shutdown How a stopping server drains, so a rolling restart does not cut running work. On SIGTERM it keeps serving for `SERVER_SHUTDOWN_DELAY_SECS` while load balancers stop routing to it, then stops accepting, ends event streams with a `disconnecting` event (reason `shutdown`) so clients reconnect to another replica, and lets in-flight requests finish for up to `SERVER_SHUTDOWN_GRACE_SECS`. The worker connection closes last, after at most 5 more seconds. | Variable | Default | Description | | ---------------------------- | ------- | ------------------------------------------------------------ | | `SERVER_SHUTDOWN_DELAY_SECS` | `5` | Seconds to keep serving after SIGTERM before draining starts | | `SERVER_SHUTDOWN_GRACE_SECS` | `20` | Seconds in-flight requests get to finish once draining | **Notes:** * Keep delay + grace + 5 seconds under the orchestrator’s kill timeout (Kubernetes `terminationGracePeriodSeconds`, default 30; Compose `stop_grace_period`) * Workers drain separately: `WORKER_SHUTDOWN_GRACE_SECS` bounds how long a stopping worker waits, and running turns are handed to another worker at their next step ## VALKEY_URL Connection URL for Valkey (Redis-compatible) used for distributed rate limiting across control-plane instances. | Property | Value | | ------------ | --------------------------------------------------- | | **Required** | No | | **Default** | Not set (uses per-instance in-memory rate limiting) | **Example:**
```bash
# Local Valkey
VALKEY_URL=redis://localhost:6379
# With authentication
VALKEY_URL=redis://user:password@valkey.example.com:6379
# TLS (managed cloud service)
VALKEY_URL=rediss://user:password@valkey.example.com:6380
```
**Notes:** * When not set, rate limiting falls back to in-memory governor (per-instance, no coordination) * With N instances behind a load balancer, per-instance rate limiting allows N× the intended budget per IP, set `VALKEY_URL` for coordinated limits * Accepts `redis://`, `rediss://` (TLS), `valkey://`, `valkeys://` (TLS) schemes * Fail-open: if Valkey is unreachable, requests are allowed (availability over strictness) * Only used by control-plane (server); workers don’t need this variable * Uses sliding-window counters via Lua scripts for atomic rate limit checks ## DATABASE_URL PostgreSQL connection URL for the control plane. Workers do not connect to the database; they reach it through the API over gRPC. | Property | Value | | ------------ | ------------------------------------------------------------------------- | | **Required** | Yes, unless `DEV_MODE=true` | | **Default** | Not set (startup fails with `DATABASE_URL environment variable required`) | **Example:**
```bash
DATABASE_URL=postgres://everruns:secret@db.example.com:5432/everruns?sslmode=require
```
**Notes:** * Startup warns when the URL has no `sslmode` or uses `sslmode=disable`. * When this points at a pooler, set [`DATABASE_UNPOOLED_URL`](#database_unpooled_url) for listeners. ## DATABASE_UNPOOLED_URL Direct PostgreSQL connection URL used only for session-scoped `LISTEN/NOTIFY` listeners. | Property | Value | | ------------ | --------------------------------------------------------------------- | | **Required** | No | | **Default** | Not set (listeners reuse `DATABASE_URL` if it is a direct connection) | **Example:**
```bash
# Query traffic through a pooler, listeners through a direct endpoint
DATABASE_URL=postgres://app:secret@ep-foo-pooler.us-east-1.aws.neon.tech/everruns?sslmode=require
DATABASE_UNPOOLED_URL=postgres://app:secret@ep-foo.us-east-1.aws.neon.tech/everruns?sslmode=require
```
**Notes:** * Use this when `DATABASE_URL` points at Neon `-pooler`, PgBouncer, or another proxy that does not preserve session-scoped `LISTEN/NOTIFY` semantics. * Listener paths include PostgreSQL-backed event wakeups, notification SSE, and PG task notification fallback when NATS is unavailable. * If `DATABASE_URL` or `DATABASE_UNPOOLED_URL` appears to point at a pooled/proxied endpoint, startup now fails fast with guidance to set a direct listener URL. * Ordinary query traffic still uses `DATABASE_URL`. ## Object Storage (S3-compatible blob backend) Optional backend that offloads workspace-file and image *content bytes* to an S3-compatible object store while keeping all metadata in PostgreSQL. Everruns remains the proxy for every read/write, no S3 presigned URLs are handed to clients or workers. | Variable | Required | Default | Description | | ------------------------------------- | --------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `STORAGE_BLOB_BACKEND` | No | `db` | `db` keeps bytes inline in PostgreSQL (current behavior); `s3` offloads to object storage. | | `STORAGE_S3_BUCKET` | When `s3` | , | Target bucket name. | | `STORAGE_S3_REGION` | No | , | Bucket region / region label. | | `STORAGE_S3_ENDPOINT` | No | , | Custom endpoint for S3-compatible stores (SeaweedFS, R2). Unset for AWS S3. | | `STORAGE_S3_ACCESS_KEY_ID` | No | , | Static access key. Omit to use the AWS credential chain (IAM role/instance). | | `STORAGE_S3_SECRET_ACCESS_KEY` | No | , | Static secret key. | | `STORAGE_S3_PREFIX` | No | (empty) | Key prefix isolating multiple deployments within one bucket. | | `STORAGE_S3_ALLOW_HTTP` | No | `false` | Allow plaintext HTTP (local/dev only, e.g. SeaweedFS over HTTP). | | `STORAGE_S3_FORCE_PATH_STYLE` | No | `true` | Use path-style requests (required by SeaweedFS; harmless on AWS S3). | | `STORAGE_BLOB_GC_INTERVAL_SECONDS` | No | `21600` (6h) | Interval between blob GC sweeps that reclaim orphaned objects. `0` disables GC. Only effective with the `s3` backend (inline `db` storage has no orphans). | | `STORAGE_BLOB_GC_GRACE_SECONDS` | No | `86400` (24h) | Safety grace period; orphaned objects younger than this are never deleted (avoids racing in-flight creates). | | `STORAGE_BLOB_GC_MAX_DELETES_PER_RUN` | No | `10000` | Per-sweep deletion cap to bound work; remaining orphans are reclaimed next sweep. | | `STORAGE_BLOB_GC_MAX_LIST_PER_RUN` | No | `100000` | Per-sweep cap on objects listed per prefix, bounding GC memory; larger buckets are reconciled across sweeps in key-order windows. | | `SESSION_FILE_SINGLE_MAX_BYTES` | No | 100 MiB (`db`), 2 GiB (`s3`) | Largest single file a session or workspace may hold. | | `SESSION_FILE_MAX_BYTES` | No | 500 MiB (`db`), 10 GiB (`s3`) | Total file bytes per session or workspace. | **Example (local SeaweedFS via `just seaweedfs`):**
```bash
STORAGE_BLOB_BACKEND=s3
STORAGE_S3_BUCKET=everruns-dev
STORAGE_S3_ENDPOINT=http://127.0.0.1:8333
STORAGE_S3_REGION=us-east-1
STORAGE_S3_ACCESS_KEY_ID=everruns
STORAGE_S3_SECRET_ACCESS_KEY=everruns-secret
STORAGE_S3_ALLOW_HTTP=true
```
Any S3-compatible store works (AWS S3, SeaweedFS, R2); only the endpoint and credentials differ. **Notes:** * The backend is selected per process at startup; a deployment runs entirely on `db` or `s3`. Enabling `s3` offloads newly written content; pre-existing inline content is still served transparently. * Tenant isolation is by object key (`workspaces/{workspace_id}/…`, `images/org-{org_id}/…`) plus the existing org/workspace authorization layer. ## NATS_URL Connection URL for Core NATS, used for push-based event delivery and task notifications. Set it on the control plane only. Workers never connect to NATS: the server tells them whether streamed output can skip PostgreSQL. | Property | Value | | ------------ | ------------------------------------------------------------------------------------------- | | **Required** | With more than one control-plane instance | | **Default** | Not set (uses PG NOTIFY for task notifications, in-memory broadcast for SSE event delivery) | **Example:**
```bash
# Local NATS
NATS_URL=nats://localhost:4222
# Cluster
NATS_URL=nats://nats1:4222,nats://nats2:4222,nats://nats3:4222
# Server with `authorization { users: [...] }`
NATS_URL=nats://control:s3cret@nats:4222
```
**Notes:** * When not set, all events persist to PG, task notifications use PG NOTIFY, and live session events are pushed inside the one process. That covers a single control-plane instance only: with `EXPECTED_INSTANCES` above 1 and no `NATS_URL`, each instance polls PostgreSQL for live events instead and reports `degraded` in `/health`. * When set, enables two features: * **Ephemeral event delivery**: delta events (`output.message.delta`, `reason.thinking.delta`, `tool.output.delta`, and voice transcript deltas) skip PostgreSQL and flow only through Core NATS. `llm.generation` remains durable in PostgreSQL. SSE streams subscribe to NATS instead of polling PG. * **Task notifications**: `task.available.{activity_type}` subjects replace PG NOTIFY for push-based worker notification. Lower latency (\~1ms vs \~30ms), supports multi-instance deployments. * JetStream is not required. Live events use Core NATS without a replay buffer; durable event replay comes from PostgreSQL. * Credentials embedded in the URL (`nats://user:password@host`) are sent as user/password auth. Reserved characters in the password (`/`, `@`, `:`, `%`) are accepted as-is, so a generated secret can be pasted unmodified; percent-encoded passwords are decoded and also work * If the NATS connection fails at startup, the server logs the connection error, uses PG NOTIFY for task notifications, and polls PostgreSQL for live session events so other instances’ events still reach its clients. `/health` reports `status: degraded` with `event_delivery: polling`, and the system doctor warns. The server retries the configured NATS connection in the background; after recovery it switches event delivery back to NATS and closes existing SSE streams so clients reconnect with their durable cursor. * Only used by control-plane (server); workers communicate via gRPC and don’t need NATS access * Default port: 4222 (or `22` with `PORT_PREFIX`, for example `27122` for prefix `271`) * `just start-all` automatically starts NATS and exports `NATS_URL` if `nats-server` is installed ## LLM Provider API Keys LLM provider API keys (OpenAI, Anthropic, Gemini) are primarily stored encrypted in the database and managed via the Models > Providers UI. | Property | Value | | ----------------------- | -------------------------------------------- | | **Storage** | Database (encrypted with AES-256-GCM) | | **Configuration** | Models > Providers UI or `/v1/providers` API | | **Supported Providers** | OpenAI, Anthropic, Google Gemini | **Required for encryption:** The `SECRETS_ENCRYPTION_KEY` environment variable must be set for the control-plane API to encrypt/decrypt API keys. Workers receive decrypted API keys via gRPC and do not need this variable.
```bash
# Generate a new key
python3 -c "import os, base64; print('kek-v1:' + base64.b64encode(os.urandom(32)).decode())"
# Set in environment (control-plane only)
SECRETS_ENCRYPTION_KEY=kek-v1:your-generated-key-here
```
### Default API Keys (Development Convenience) For development, you can set default API keys via environment variables on the **control-plane only**. These are used as fallbacks when providers don’t have keys configured in the database. | Variable | Description | | --------------------------- | --------------------------------------------- | | `DEFAULT_OPENAI_API_KEY` | Fallback API key for OpenAI providers | | `DEFAULT_ANTHROPIC_API_KEY` | Fallback API key for Anthropic providers | | `DEFAULT_GEMINI_API_KEY` | Fallback API key for Google Gemini providers | | `DEFAULT_META_API_KEY` | Fallback API key for Meta Model API providers | **Example:**
```bash
# Set in .env or environment (control-plane only)
DEFAULT_OPENAI_API_KEY=sk-...
DEFAULT_ANTHROPIC_API_KEY=sk-ant-...
DEFAULT_GEMINI_API_KEY=AIza...
DEFAULT_META_API_KEY=...
```
**Notes:** * These variables are only used by the control-plane, not workers * Workers receive API keys via gRPC from the control-plane * Database-stored keys always take priority over environment variables * These are intended for development convenience, not production use * The `just start-all` command automatically sets these from `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, and `GEMINI_API_KEY` if present * If no API key is configured for a provider, LLM calls will fail and users will see an error message in the chat: “I encountered an error while processing your request. Please try again later.” ## System Model Keys Two deployment-owned models sit outside the provider system above. Neither is selectable by an agent, neither is stored in the database, and neither is reachable from session or agent configuration — they are host services the platform uses for its own internal work. | Variable | Powers | Unset means | | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `UTILITY_OPENAI_API_KEY` | Agent Analyze/Health checks, and guardrail checks with `engine: "utility_llm"` (the default), called directly against OpenAI | Those checks are skipped; Analyze and Health are unavailable | | `UTILITY_OPENROUTER_API_KEY` | The same work, routed through OpenRouter instead. Setting it selects OpenRouter; it wins when both keys are set, and the startup log says so | The utility LLM falls back to `UTILITY_OPENAI_API_KEY` | | `UTILITY_LLM_MODEL` | The model the utility LLM calls on whichever backend was selected | Defaults to `gpt-6-luna` on OpenAI, `openai/gpt-6-luna` on OpenRouter | | `UTILITY_TYPESAFE_API_KEY` | Guardrail checks with `engine: "jev"`, answered by TypeSafe | Those checks are skipped with a warning and the turn proceeds, unless `UTILITY_DECISION_DRIVER` picks another driver | | `UTILITY_AZURE_AI_API_KEY`, `UTILITY_AZURE_AI_ENDPOINT` | Microsoft-Decision-1 on Microsoft Foundry for `jev` checks: the resource key and its resource or project endpoint (`https://.services.ai.azure.com`). Used only when `UTILITY_DECISION_DRIVER=mai` | The `mai` driver is unavailable. Setting only one of the two stops startup | | `UTILITY_GEMINI_API_KEY` | Gemini Decision 1 on Google’s Decisions API for `jev` checks. Used only when `UTILITY_DECISION_DRIVER=gemini` | The `gemini` driver is unavailable | | `UTILITY_DECISION_DRIVER` | Which decision driver answers `jev` checks and the Slack respond-or-not check: `typesafe`, `openai` (OpenAI’s Decisions API, using `UTILITY_OPENAI_API_KEY`), `openrouter` (Jev through `UTILITY_OPENROUTER_API_KEY`), `mai` (Microsoft-Decision-1 on Foundry, using `UTILITY_AZURE_AI_*`), `gemini` (Gemini Decision 1, using `UTILITY_GEMINI_API_KEY`), or `llm` to answer them with the utility LLM when you have no TypeSafe key. `llm` answers are labels, not calibrated probabilities | `typesafe` when its key is set, otherwise disabled. A driver that is not configured stops startup. Replaces `DECISIONS_DRIVER`, which now stops startup | | `UTILITY_DECISION_MODEL` | The model the chosen decision driver is asked for. For `mai` it is the Foundry deployment name | The driver’s own default (`Microsoft-Decision-1` for `mai`, `gemini-decision-1` for `gemini`). Not allowed with `UTILITY_DECISION_DRIVER=llm`, which uses `UTILITY_LLM_MODEL`. Replaces `DECISIONS_MODEL` | The keys are read from the process environment at startup. Missing keys **fail open**: a guardrail whose engine is not configured never blocks, so a missing key weakens policy rather than wedging traffic. Check the startup logs if a configured guardrail appears to do nothing. There is no separate provider variable: the utility LLM’s backend is whichever key you supply. An OpenRouter model id is namespaced by its upstream provider, so override `UTILITY_LLM_MODEL` with an id that backend accepts (`anthropic/claude-sonnet-5.5`, not `claude-sonnet-5.5`).
```bash
# Control-plane and workers both read these.
UTILITY_OPENAI_API_KEY=sk-...
# ...or route the utility LLM through OpenRouter instead:
UTILITY_OPENROUTER_API_KEY=sk-or-...
UTILITY_LLM_MODEL=openai/gpt-6-luna
UTILITY_TYPESAFE_API_KEY=ts-...
# ...or answer jev checks with OpenAI's Decisions API:
UTILITY_DECISION_DRIVER=openai
```
Agents can also be given the TypeSafe capability directly, which is a **separate** credential: a per-user connection configured in Settings > Connections, never this deployment key. See [TypeSafe](https://everruns.com/docs/integrations/typesafe/) and [Guardrails](https://everruns.com/docs/capabilities/guardrails/). ## Slack Manager App With a Slack manager app configured, an organization administrator connects a Slack workspace by approving it once in Slack, instead of pasting an app configuration token. Everruns then creates each Agent’s Slack app in that workspace through the approval. | Variable | Description | | --------------------------------- | ------------------------------------ | | `SLACK_MANAGER_APP_CLIENT_ID` | Client ID of your Slack manager app. | | `SLACK_MANAGER_APP_CLIENT_SECRET` | Client secret of the same app. | Set both, or neither; one alone is ignored with a warning. The feature also needs `SECRETS_ENCRYPTION_KEY`. Configure the Slack app as follows: * User token scopes `app_configurations:read` and `app_configurations:write`; no bot scopes. * Redirect URL `/v1/slack/workspaces/oauth/callback`, for example `https://app.example.com/api/v1/slack/workspaces/oauth/callback`. * Token rotation turned on. Everruns refuses a grant that does not expire. Slack only lets an app request the `app_configurations` scopes once it has enabled manager app support for that app. Until then, leave these unset: the configuration token path works without them and stays available beside the approval. ## System Email Delivery System email delivery is an internal service used by product and operational flows. It is not an agent capability, public API, or UI setting. | Variable | Required | Default | Description | | --------------------- | ------------------------------------- | ------------------------ | ---------------------------------------------------------------- | | `EMAIL_PROVIDER` | Yes, when sending email in production | unset / disabled | Email provider. Supported values: `disabled`, `resend` | | `RESEND_API_KEY` | Yes, when `EMAIL_PROVIDER=resend` | unset | Resend API key | | `RESEND_API_BASE_URL` | No | `https://api.resend.com` | Resend API base URL override for tests or controlled deployments | **Example:**
```bash
EMAIL_PROVIDER=resend
RESEND_API_KEY=re_...
```
**Notes:** * Set these on the control-plane process that performs system email sends. * The sender is fixed in code as `Everruns `. * The Resend account must have `everruns.com` verified and enabled for sending. ## MCP HTTP response limits `EVERRUNS_MCP_MAX_RESPONSE_BYTES` caps each successful MCP HTTP response at 16 MiB (16777216 bytes) by default. Set a positive byte count on the API, workers, and embedded host process when tools or resources legitimately return larger payloads. Invalid or zero values fail ordinary MCP requests with a configuration error.
```bash
EVERRUNS_MCP_MAX_RESPONSE_BYTES=33554432
```
The limit applies while reading JSON and SSE bodies, including initialization, resource operations, task polling, and continuation responses, even without `Content-Length`. Exceeding it aborts the response instead of parsing partial content. HTTP errors retain at most the first 8 KiB of body bytes; their status and headers survive truncation, invalid UTF-8, and body failures. Existing request timeouts still bound stalled responses. Tool discovery retains its separate catalog budgets. ## Outbound egress These apply to tenant traffic: requests an agent, capability, MCP server or integration chooses. Platform traffic (LLM providers, system email) is not affected. Set the same values on the control plane and workers. See [Outbound traffic](https://everruns.com/docs/advanced/physical-architecture/#outbound-traffic). | Variable | Default | Description | | --------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `EVERRUNS_EGRESS_POLICY` | `open` | `open`, `curated-writes` or `curated-all`. An unrecognized value enforces `curated-all`. | | `EVERRUNS_EGRESS_OPEN_READS_PER_MINUTE` | `120` | Per-org, per-process limit on reads to hosts outside the allowlist in `curated-writes`. `0` turns those reads off. | | `EVERRUNS_EGRESS_REPUTATION` | `off` | `cloudflare-security` checks hosts against Cloudflare’s security resolver before connecting; lookups that fail let the request through. | | `EVERRUNS_EGRESS_DENY_FEEDS` | unset | Comma-separated threat-intelligence domain lists to refuse: `hagezi-tif-mini` (HaGeZi Threat Intelligence Feeds, mini) or your own `https://` URLs of plain, hosts-file or adblock-style domain lists. A listed domain and its subdomains are refused in every mode and for every org, open egress included, unless the curated allowlist names the host. Fetched at start and every 6 hours; a failed fetch keeps the last list. | | `EVERRUNS_EGRESS_PROXY_URL` | unset | Send tenant traffic through a tenant egress proxy, for example `http://egress-proxy:4750`. | | `EVERRUNS_EGRESS_PROXY_SECRET` | unset | Shared with the proxy, at least 32 bytes. With a proxy URL and no usable secret, tenant requests fail rather than leave directly. | | `EVERRUNS_EGRESS_PROXY_SOCKS_ADDRESS` | unset | Public `host:port` of the proxy’s SOCKS5 listener. When set, E2B sandboxes send their own outbound TCP through the proxy (E2B’s bring-your-own-proxy), under the same checks, caps and per-org counts as tenant traffic. Needs `EVERRUNS_EGRESS_PROXY_SECRET`; if it is missing or short, E2B sandboxes fail to start rather than leave directly. Isolated sandboxes do not need it: they reach the listener inside the cluster with credentials signed by `EVERRUNS_EGRESS_PROXY_SECRET`. | | `EVERRUNS_EGRESS_PROXY_SOCKS_TLS` | `true` | E2B connects to the SOCKS5 address over TLS, so put a TLS terminator with a publicly trusted certificate in front of the listener. Set `false` only when E2B reaches the proxy over a private network: the sandbox credential would otherwise cross the internet in the clear. | ## Isolated sandboxes Settings of the [Isolated sandbox](https://everruns.com/docs/features/isolated-sandboxes/) target, set on the API and the workers alike. The Helm chart sets them from `sandboxes.kubernetes`. | Variable | Default | Effect | | ----------------------------------------------------------------------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `KUBERNETES_SANDBOX_NAMESPACE` | unset | Namespace sandbox pods run in. Turns the target on. | | `KUBERNETES_SANDBOX_RUNTIME_CLASS` | unset | Runtime class of sandbox pods: `isolated-sandbox` (AWS Nitro Sandboxes on EKS), `gvisor` or `kata`. Unset runs plain containers, which the UI flags as no security boundary. | | `KUBERNETES_SANDBOX_IMAGE` | `ghcr.io/everruns/sandbox-dev:v` | Image of templates that name none. | | `KUBERNETES_SANDBOX_DESKTOP_IMAGE` | `ghcr.io/everruns/sandbox-desktop:v` | Image of desktop sandboxes whose template names none. | | `KUBERNETES_SANDBOX_ALLOWED_IMAGES` | any | Comma-separated image prefixes templates may use. | | `KUBERNETES_SANDBOX_STORAGE_CLASS` | cluster default | Storage class of workspace volumes. | | `KUBERNETES_SANDBOX_WORKSPACE_SIZE` | `10Gi` | Size of each workspace volume. | | `KUBERNETES_SANDBOX_NODE_SELECTOR`, `KUBERNETES_SANDBOX_TOLERATIONS` | none | JSON node selector and tolerations for sandbox pods. | | `KUBERNETES_SANDBOX_SIZES` | `small,medium,large` | Sizes templates may pick. | | `KUBERNETES_SANDBOX_MAX_PER_ORG` | `5` | Running sandboxes per organization; paused ones do not count. | | `KUBERNETES_SANDBOX_BILLING_NOTE` | unset | What a running sandbox costs the organization, in plain words. Shown where a template picks Isolated sandbox. Leave unset when sandboxes are free. | | `KUBERNETES_SANDBOX_EGRESS` | `proxy` with a proxy address, else `none` | Where open-network sandboxes may connect: `proxy`, `direct` (internet, private ranges excluded) or `none`. A wrong value stops the server rather than widen egress. | | `KUBERNETES_SANDBOX_EGRESS_PROXY_ADDRESS` | unset | In-cluster `host:port` of the egress proxy’s SOCKS5 listener. | | `KUBERNETES_SANDBOX_EGRESS_PROXY_POD_LABELS`, `KUBERNETES_SANDBOX_EGRESS_PROXY_NAMESPACE` | required with `proxy` | Labels (`key=value,...`) and namespace of the proxy pods the network policy lets sandboxes reach. | | `KUBERNETES_SANDBOX_GUEST_ROUTE` | `pod_ip` in a cluster, else `api_proxy` | How the server reaches the guest: straight to the pod IP, or through the API server’s pod proxy. | | `KUBERNETES_SANDBOX_CONTROL_POD_LABELS`, `KUBERNETES_SANDBOX_CONTROL_NAMESPACE` | any pod | Pods allowed to reach the guest port. Every request also needs the sandbox’s own token. | | `KUBERNETES_SANDBOX_READY_TIMEOUT_SECONDS` | `300` | How long a new pod may take to start. | ## Tenant egress proxy Read by the `everruns-egress-proxy` binary. Every cap is off unless set. | Variable | Default | Description | | ---------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EVERRUNS_EGRESS_PROXY_SECRET` | required | Same value as on the control plane and workers, at least 32 bytes. The proxy refuses to start without it. | | `EGRESS_PROXY_ADDR` | `0.0.0.0:4750` | Proxy listener. Keep it on the private network. | | `EGRESS_PROXY_HEALTH_ADDR` | `0.0.0.0:4751` | Health listener, `GET /healthz`. | | `EGRESS_PROXY_SOCKS_ADDR` | unset | SOCKS5 listener for sandbox providers, for example `0.0.0.0:1080`. Off unless set. It accepts only sandbox credentials and CONNECT. Expose it to E2B through a TLS terminator, and pass the public address to the control plane and workers as `EVERRUNS_EGRESS_PROXY_SOCKS_ADDRESS`. | | `EGRESS_PROXY_ORG_RPM` | no cap | Requests per minute per org. | | `EGRESS_PROXY_ORG_HOST_RPM` | no cap | Requests per minute per org to one host. | | `EGRESS_PROXY_ORG_CONCURRENCY` | no cap | Open connections per org. | | `EGRESS_PROXY_ORG_BYTES_PER_DAY` | no cap | Bytes per org per UTC day. Counted per replica unless a control endpoint reports what was already used. | | `EGRESS_PROXY_GLOBAL_PER_HOST_RPM` | no cap | Requests per minute to one host across all orgs. | | `EGRESS_PROXY_REPLICAS` | `1` | Replicas sharing the caps; each enforces its share. | | `EGRESS_PROXY_BIND_ADDR` | unset | Source IP for outbound connections, when the host has a dedicated tenant address. | | `EGRESS_PROXY_INSTANCE` | hostname | Label on usage batches. | | `EGRESS_PROXY_CONTROL_URL` | unset | Control endpoint. The proxy asks `GET /internal/egress-proxy/orgs/{org}/limits` (cached 60 s) and posts usage to `POST /internal/egress-proxy/usage`, both with `Authorization: Bearer `. The `EGRESS_PROXY_ORG_*` caps then apply only while it cannot be reached. | Usage without a control endpoint is logged once per org, session and minute on the `everruns::egress::usage` target. The proxy also counts abuse signals per org and minute: attempts on blocked (SMTP) ports, destinations that resolve to private addresses, requests refused by a cap, and how many distinct hosts the org reached. They go to the control endpoint in the same usage batches (`signals`), or are logged on the `everruns::egress::signals` target. The proxy only counts them; acting on them, such as suspending an org, is up to the control plane. ## Data plane v2 The link between the API (control plane) and its data plane, and where new organizations run. Docker Compose, the Helm chart and `everruns docker up` set these; the Compose guide’s `.env` needs the first two. | Variable | Value | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EVERRUNS_DP_TOKEN` | Bearer token the data plane and the API use with each other. Shards get the same value as `AGENTD_DP_TOKEN`. Generate with `openssl rand -hex 32`. | | `EVERRUNS_DP_KEY` | 32 random bytes in base64 that wrap each organization’s data key. Shards get the same value as `AGENTD_DP_KEY`. Generate with `openssl rand -base64 32`. Keep it: a new key cannot open what the old one wrapped. | | `EVERRUNS_DEFAULT_DATA_PLANE_URL` | The data plane’s gateway: a node of its directory shard, or a Service over its ready nodes, such as `http://shard:7000`. Any node forwards to the shard that holds an organization or session, so one address covers every shard. The API registers it and puts every organization that has never had a session on it. | | `EVERRUNS_DEFAULT_DATA_PLANE` | That data plane’s id. Default `local`. | | `EVERRUNS_DEFAULT_DATA_PLANE_SINCE` | Optional RFC 3339 time: place only organizations created at or after it, so ones that signed up earlier and never ran an agent stay on v1. A value that does not parse turns the default data plane off. | | `EVERRUNS_DATA_PLANE` | `v1` turns placement off: new organizations run on the workers. | **Example (`.env`):**
```bash
EVERRUNS_DP_TOKEN=3f1c...e9a2 # openssl rand -hex 32
EVERRUNS_DP_KEY=q8m3...Zk0= # openssl rand -base64 32
```
**Notes:** * Without the token and key, the API logs a warning and new organizations run on v1. * Organizations with v1 sessions stay on the workers. ## UI API Proxy Architecture The UI makes all REST API requests (including SSE) to `/api/*` paths. The backend serves those routes under `/api` directly. Root-level backend routes like `/oauth/*`, `/mcp`, and `/.well-known/*` bypass the UI and are proxied straight to the backend. **Local Development:** * Caddy on `:9300` routes `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to backend at `:9301` * Example: `/api/v1/agents` → `http://localhost:9301/api/v1/agents` * Example: `/oauth/authorize?...` → `http://localhost:9301/oauth/authorize?...` * Example: `/mcp` → `http://localhost:9301/mcp` * Example: `/.well-known/oauth-authorization-server` → `http://localhost:9301/.well-known/oauth-authorization-server` * SSE streaming works via `flush_interval -1` in Caddy config * No CORS needed (same-origin through Caddy) **Production:** * Configure your reverse proxy (nginx, Caddy, etc.) to route `/api/*`, `/oauth/*`, `/mcp`, and `/.well-known/*` to the API server * Disable response buffering for SSE endpoints * [Example Caddy configuration](https://github.com/everruns/everruns/blob/main/local/Caddyfile) ## Other Server Variables | Variable | Default | Description | | ------------------------------------ | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AUTH_MODE` and `AUTH_*` | Depends on `DEPLOYMENT_GRADE` | Authentication mode and its settings (`AUTH_BASE_URL`, `AUTH_JWT_SECRET`, `AUTH_LOGIN_ORIGIN`, and others). See [Authentication](https://everruns.com/docs/sre/runbooks/authentication/). | | `RATE_LIMIT_API_REQUESTS_PER_MINUTE` | `1200` | API request rate limit per minute. Uses Valkey when `VALKEY_URL` is set, otherwise a per-instance in-memory limiter. | | `TRUSTED_PROXY_HOPS` | `1` | Number of trusted reverse proxies in front of the server. The client IP is taken from `X-Forwarded-For` counting this many entries from the right. Raise it if a CDN or load balancer sits in front of your reverse proxy. | | `EVENT_RETENTION_DAYS` | `0` (disabled) | Archive session events older than this many days. A background task runs hourly. | | `DATABASE_POOL_MAX` | `50` | Maximum PostgreSQL connections in the request pool. | | `DATABASE_POOL_MIN` | `5` | Minimum PostgreSQL connections kept open. | | `SECRETS_ENCRYPTION_KEY_PREVIOUS` | Unset | Previous encryption key, kept during key rotation so existing values still decrypt. See [Encryption key rotation](https://everruns.com/docs/sre/runbooks/encryption-key-rotation/). | | `EXPECTED_INSTANCES` | `1` | Number of control-plane instances behind the load balancer. Divides the global and per-org SSE connection limits across instances. Per-session limits are not divided. Above 1 without `NATS_URL`, live session events are polled from PostgreSQL (degraded). | ## Throughput tuning These change how much database work each agent turn costs. The defaults suit most deployments; the switches exist for comparisons and for the rare deployment that needs another trade. | Variable | Default | Description | | ------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EVENT_GROUP_COMMIT` | on | Events that arrive while a commit is in flight are stored together in the next transaction, so many turns share one WAL flush. Each write is answered after its commit, so it is as durable as a lone write. `off`, `0` or `false` stores each append in its own transaction. | | `EVENT_GROUP_COMMIT_LANES` | `4` | Parallel commit lanes per API instance. A session always uses the same lane, so its events keep their order. | | `SESSION_ACTOR_STICKY_GRACE_MS` | `300` | How long a session’s next turn waits for the worker that ran its last turn, which still holds its history in memory, before any worker may take it. `0` turns this off. | ## SSE Streaming Configuration | Variable | Default | Description | | ----------------------------- | ------- | ------------------------------------------------------------------ | | `SSE_REALTIME_CYCLE_SECS` | `300` | Connection cycle interval for session event streams (seconds) | | `SSE_MONITORING_CYCLE_SECS` | `600` | Connection cycle interval for durable monitoring streams (seconds) | | `SSE_HEARTBEAT_INTERVAL_SECS` | `30` | Interval between heartbeat comments on all SSE streams (seconds) | | `SSE_GLOBAL_MAX` | `10000` | Maximum total SSE connections across all users | | `SSE_PER_SESSION_MAX` | `12` | Maximum SSE connections per session | | `SSE_PER_ORG_MAX` | `1000` | Maximum SSE connections per organization | **Notes:** * Heartbeat comments (`: heartbeat\n\n`) are sent on all SSE streams to detect stale connections * The heartbeat interval must be less than the SDK read timeout (default: 45s) with safety margin * Connection cycling prevents stale connections through proxies and load balancers * When running behind HTTP/1.1 proxies, increase `SSE_REALTIME_CYCLE_SECS` to reduce reconnection frequency ## Worker gRPC Configuration ### SERVER_GRPC_ADDRESS Address of the server gRPC endpoint for worker communication. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `127.0.0.1:9001` | **Example:**
```bash
SERVER_GRPC_ADDRESS=127.0.0.1:9001
```
**Notes:** * Workers communicate with the server via gRPC for all database operations * `WORKER_GRPC_ADDRESS` is supported as a legacy alias * The server exposes both HTTP (default `9000`) and gRPC (default `9001`) interfaces * Workers are stateless and do not connect directly to the database ### WORKER_GRPC_AUTH_TOKEN Bearer token for authenticating worker gRPC connections to the control-plane. | Property | Value | | ------------ | ------------------------------- | | **Required** | Yes (production); No (dev mode) | | **Default** | Unset (auth disabled) | **Example:**
```bash
WORKER_GRPC_AUTH_TOKEN=your-secret-token
```
**Notes:** * Must be set on both the server and all workers (same value) * When unset, gRPC auth is disabled (acceptable for local development only) * Server panics on startup if unset when not in dev mode ### SERVER_GRPC_BIND_ADDR Bind address for the server-side gRPC listener. | Property | Value | | ------------ | ---------------- | | **Required** | No (server only) | | **Default** | `0.0.0.0:9001` | **Example:**
```bash
SERVER_GRPC_BIND_ADDR=0.0.0.0:9001
```
**Notes:** * `WORKER_GRPC_ADDR` is supported as a legacy alias ### WORKER_GRPC_CONNECT_TIMEOUT Timeout in seconds for worker initial connection to control-plane gRPC. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `30` | **Example:**
```bash
WORKER_GRPC_CONNECT_TIMEOUT=60
```
### WORKER_GRPC_REQUEST_TIMEOUT_SECS Deadline in seconds for each worker call to the control-plane gRPC service. A call that gets no answer by then fails as a retryable error, and the turn resumes from the session log instead of hanging. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `120` | **Example:**
```bash
WORKER_GRPC_REQUEST_TIMEOUT_SECS=120
```
### WORKER_ACTOR_STALL_TIMEOUT_SECS How long, in seconds, a worker may hold a session while making no progress before its turn starts or after it ends. Past it the worker lets the session go, and another worker runs the turn; a session that keeps stalling is stopped with a processing error. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `300` | **Example:**
```bash
WORKER_ACTOR_STALL_TIMEOUT_SECS=300
```
### WORKER_TOOL_DETACH_AFTER_SECS How long, in seconds, a server-side tool call may run inside its turn. A call still running after that keeps running on the worker, and its turn gives back its memory and session lease until the call finishes. `0` keeps every call inside its turn. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `300` | **Example:**
```bash
WORKER_TOOL_DETACH_AFTER_SECS=300
```
### WORKER_MAX_RUNNING_TURNS Most session turns one worker runs at once. Turns spend most of their time waiting on the model and on tools, so they do not use the worker’s task slots (`MAX_CONCURRENT_TASKS`). Unset or `0` derives it from the memory limit: 85% of it over about 2 MiB per turn, between 100 and 10000, or 2000 when the limit is unknown. A 6 GiB worker runs up to about 2600 turns. Before that ceiling, a worker running at least 32 turns stops taking new ones while its resident memory is over 85% of its limit, or while its async runtime lags (see `WORKER_MAX_RUNTIME_LAG_MS`). Running turns are never stopped. | Property | Value | | ------------ | --------------------- | | **Required** | No (worker only) | | **Default** | From the memory limit | **Example:**
```bash
WORKER_MAX_RUNNING_TURNS=2000
```
### WORKER_MEMORY_LIMIT_MIB The memory, in MiB, that `WORKER_MAX_RUNNING_TURNS` and the memory brake are measured against. Unset, the worker reads its cgroup limit (Kubernetes and Docker memory limits); without one there is no memory brake. | Property | Value | | ------------ | ----------------------- | | **Required** | No (worker only) | | **Default** | The cgroup memory limit | **Example:**
```bash
WORKER_MEMORY_LIMIT_MIB=6144
```
### WORKER_MAX_RUNTIME_LAG_MS How late, in milliseconds, the worker’s async runtime may run a timer (a moving average) before the worker stops taking new turns. CPU-heavy turns, such as in-process interpreters and large prompts, show up here first. `0` turns the brake off. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `25` | **Example:**
```bash
WORKER_MAX_RUNTIME_LAG_MS=25
```
### MAX_DETACHED_TOOL_CALLS Most tool calls one worker keeps running after their turns parked. Past it a long call stays inside its turn. | Property | Value | | ------------ | ---------------- | | **Required** | No (worker only) | | **Default** | `1000` | **Example:**
```bash
MAX_DETACHED_TOOL_CALLS=1000
```
### WORKER_GRPC_TLS_CERT Path to PEM-encoded certificate file. On the server, this is the gRPC server certificate. On the worker, this is the client certificate presented during mTLS handshake. | Property | Value | | ------------ | ---------------------- | | **Required** | No | | **Default** | Not set (TLS disabled) | **Example:**
```bash
WORKER_GRPC_TLS_CERT=/etc/everruns/grpc-cert.pem
```
**Notes:** * Must be set together with `WORKER_GRPC_TLS_KEY` * Server: enables TLS on the gRPC listener when both cert and key are set * Worker: presents client certificate to the server when both cert and key are set (requires `WORKER_GRPC_TLS_CA_CERT`) ### WORKER_GRPC_TLS_KEY Path to PEM-encoded private key file corresponding to `WORKER_GRPC_TLS_CERT`. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:**
```bash
WORKER_GRPC_TLS_KEY=/etc/everruns/grpc-key.pem
```
### WORKER_GRPC_TLS_CA_CERT Path to PEM-encoded CA certificate bundle for verifying the remote peer. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:**
```bash
WORKER_GRPC_TLS_CA_CERT=/etc/everruns/grpc-ca.pem
```
**Notes:** * Server: when set, requires workers to present valid client certificates signed by this CA (mutual TLS) * Worker: when nonempty, both the durable/task client and the turn client (including raw blob fetches) verify the server’s certificate against this CA and use `https://`. TLS keeps the original hostname instead of the plaintext replica balancer. Verification failures do not fall back to plaintext. * For full mTLS, set on both server and worker alongside their respective cert/key pairs * The turn client’s connection and TLS handshake are bounded by `WORKER_GRPC_REQUEST_TIMEOUT_SECS`. Unreadable TLS files fail startup; invalid certificates or mismatched identities refuse the connection. * On upgrade, check that the configured CA, expected server name and optional client identity work for turn RPCs as well as task RPCs: older workers applied these settings only to the durable/task client. Unset or empty worker CA configuration retains plaintext transport. ### WORKER_GRPC_TLS_DOMAIN Override the expected server domain name for TLS certificate verification (worker only). | Property | Value | | ------------ | ------------------------------------------- | | **Required** | No | | **Default** | Derived from `SERVER_GRPC_ADDRESS` hostname | **Example:**
```bash
WORKER_GRPC_TLS_DOMAIN=control-plane.internal
```
**Notes:** * Useful when the server certificate CN/SAN differs from the connection hostname (e.g., connecting via IP but cert has a DNS name) ## OpenTelemetry Configuration Everruns supports distributed tracing via OpenTelemetry with OTLP export. Agent traces follow the OpenTelemetry [Gen-AI agent and inference conventions](https://github.com/open-telemetry/semantic-conventions-genai/tree/main/docs/gen-ai) and, on the same spans, the [OpenInference conventions](https://arize-ai.github.io/openinference/spec/semantic_conventions.html) read by Arize Phoenix, so one OTLP endpoint serves both families of backends. ### OTEL_EXPORTER_OTLP_ENDPOINT OTLP endpoint for trace export (e.g., Grafana Tempo, Arize Phoenix, Datadog, or any OTLP-compatible backend). | Property | Value | | ------------ | -------------------------- | | **Required** | No | | **Default** | Not set (tracing disabled) | **Example:**
```bash
# For a local OTLP collector or Phoenix
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# For production Tempo
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.monitoring:4318
```
**Notes:** * When set, traces are exported via OTLP over HTTP/protobuf (use the backend’s HTTP port, typically 4318, not the gRPC port 4317) * Point this at the base endpoint; the `/v1/traces` path is appended for you, and a full signal URL is used as given * Connect to any OTLP-compatible backend for trace visualization * See [OpenTelemetry](https://everruns.com/docs/observability/opentelemetry/) for the span model and attributes * Without this variable, only console logging is enabled ### OTEL_SERVICE_NAME Service name for traces. | Property | Value | | ------------ | --------------------------------------------------- | | **Required** | No | | **Default** | `everruns-server` (API), `everruns-worker` (Worker) | **Example:**
```bash
OTEL_SERVICE_NAME=everruns-prod-api
```
### OTEL_SERVICE_VERSION Service version for traces. | Property | Value | | ------------ | --------------------- | | **Required** | No | | **Default** | Cargo package version | ### OTEL_ENVIRONMENT Deployment environment label. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | Not set | **Example:**
```bash
OTEL_ENVIRONMENT=production
```
### OTEL_RECORD_CONTENT Enable recording of LLM input/output content in traces. **Warning:** May contain sensitive data. | Property | Value | | ------------ | ------- | | **Required** | No | | **Default** | `false` | **Example:**
```bash
# Standard OTel env var (preferred)
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
# Legacy alias (also works)
OTEL_RECORD_CONTENT=true
```
**Notes:** * When enabled, the chat span records `gen_ai.system_instructions`, `gen_ai.input.messages`, `gen_ai.output.messages`, and `gen_ai.tool.definitions` (plus the OpenInference `input.value`, `output.value`, and flattened `llm.input_messages.*`); tool spans record `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` (plus `everruns.tool.executed_arguments` when a pre-tool hook rewrote the call); the turn root records the input message and final answer; the thinking span records the reasoning text * Disabled by default for privacy and data size concerns * Only enable in development or when debugging specific issues ### EVERRUNS_TRACE_CONVENTIONS Which attribute vocabularies agent spans carry. | Property | Value | | ------------ | ---------------------- | | **Required** | No | | **Default** | `gen_ai,openinference` | **Example:**
```bash
# Only the OpenTelemetry Gen-AI attributes (Tempo, Jaeger, Datadog, Langfuse)
EVERRUNS_TRACE_CONVENTIONS=gen_ai
# Only the OpenInference attributes (Arize Phoenix)
EVERRUNS_TRACE_CONVENTIONS=openinference
```
**Notes:** * Span names, kinds, and hierarchy are the same under either vocabulary; only attributes differ * Unknown values are ignored, and an empty selection falls back to both ## Monitoring detail Monitoring records one-minute history in PostgreSQL with no setup. These two variables add optional 10-second, per-node detail for data plane v2 shards, read from a Prometheus-compatible store that scrapes the shards’ `/metrics`. | Variable | Value | | ----------------------- | ------------------------------------------------------------------------------ | | `MONITORING_DETAIL` | `prometheus` (also `victoriametrics`) to read detail; unset or `off` for none. | | `MONITORING_DETAIL_URL` | Base URL of the store’s query API, `http` or `https`. |
```bash
# Docker Compose with the metrics profile
MONITORING_DETAIL=prometheus
MONITORING_DETAIL_URL=http://victoriametrics:8428
```
**Notes:** * The address is read from the environment only, never set through the API. * Pages show only the store’s scheme and host, never credentials or a path. * **System → Health** reports whether the store answers. ## Local Development with OpenTelemetry To visualize traces locally, point `OTEL_EXPORTER_OTLP_ENDPOINT` at any OTLP-compatible collector:
```bash
# Set OTLP endpoint for API and Worker
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# Start services
just start-all
```
To see traces in Arize Phoenix, run Phoenix locally and point the same variable at its OTLP/HTTP port (`http://localhost:6006`); spans render as AGENT, LLM, and TOOL spans with token counts and, when content capture is on, the messages. ### Gen-AI Trace Structure Traces follow the agentic execution lifecycle with 13 event types; every span starts and ends at the timestamp of the event it records:
```plaintext
invoke_agent {agent name} (root span, INTERNAL)
├── reason (LLM reasoning phase)
│ └── chat {model} (LLM API call, CLIENT)
│ └── thinking (extended thinking, if enabled)
├── act (tool execution phase)
│ ├── execute_tool {name}
│ └── execute_tool {name}
├── reason (iteration 2)
│ └── chat {model}
└── ...
```
### Gen-AI Trace Attributes Spans carry the OpenTelemetry Gen-AI attributes and the OpenInference attributes side by side (see `EVERRUNS_TRACE_CONVENTIONS`). The most useful ones: | Attribute | Span Types | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `gen_ai.operation.name` | invoke_agent, chat, execute_tool | `invoke_agent`, `chat`, or `execute_tool` | | `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.description` | invoke_agent | Agent the turn runs as | | `gen_ai.conversation.id` / `session.id` | All | Session identifier | | `gen_ai.provider.name` / `llm.provider` | chat | Provider (`openai`, `anthropic`, `gcp.gemini`, `aws.bedrock`, …) | | `gen_ai.request.model`, `gen_ai.response.model` / `llm.model_name` | chat | Model name | | `gen_ai.response.id` | chat | Provider response identifier | | `gen_ai.response.finish_reasons` | chat | Why generation stopped (string array) | | `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` / `llm.token_count.*` | chat, invoke_agent | Token usage (turn root carries the cumulative total) | | `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens` | chat, invoke_agent | Prompt-cache tokens | | `gen_ai.request.temperature`, `gen_ai.request.max_tokens`, `gen_ai.request.reasoning.level`, `gen_ai.request.stream` / `llm.invocation_parameters` | chat | Request parameters | | `gen_ai.response.time_to_first_chunk` | chat | Streaming latency in seconds | | `gen_ai.conversation.compacted` | chat | Context was compacted before the call | | `llm.cost.total` / `everruns.usage.cost_usd` | chat, invoke_agent | Cost in USD when known | | `gen_ai.tool.name`, `gen_ai.tool.call.id`, `gen_ai.tool.type`, `gen_ai.tool.description` / `tool.name`, `tool.description` | execute_tool | Tool identity | | `openinference.span.kind` | All | `AGENT`, `LLM`, `TOOL`, or `CHAIN` | | `error.type` | All | Low-cardinality error class on failure (error code, HTTP status, `timeout`, or `_OTHER`); the span status carries the message | | `everruns.phase` | reason, act, thinking | Phase span marker | | `everruns.turn.id`, `everruns.exec.id` | All | Everruns correlation ids | | `everruns.turn.iterations`, `everruns.turn.tool_call_count`, `everruns.turn.llm_call_count` | invoke_agent | Turn counters | | `everruns.tool.status` | execute_tool | `success`, `error`, `timeout`, or `cancelled` | ## Braintrust Integration Everruns supports sending turn, reasoning, tool, and session lifecycle events to [Braintrust](https://www.braintrust.dev/) for observability, evaluation, and logging. For setup instructions and configuration details, see the [Braintrust Integration Guide](https://everruns.com/docs/observability/braintrust/). | Variable | Required | Default | Description | | -------------------------------- | -------- | ------------------------------- | --------------------------------------------------------------- | | `BRAINTRUST_ENABLED` | No | enabled when API key is present | Explicit Braintrust on/off switch | | `BRAINTRUST_API_KEY` | Yes | - | API key from Braintrust settings | | `BRAINTRUST_PROJECT_NAME` | No | `My Project` | Project name for organizing traces | | `BRAINTRUST_PROJECT_ID` | No | - | Direct project UUID (skips name lookup) | | `BRAINTRUST_API_URL` | No | `https://api.braintrust.dev` | API base URL | | `BRAINTRUST_QUEUE_CAPACITY` | No | `1024` | Buffered event capacity before new exports are dropped | | `BRAINTRUST_MAX_BATCH_SIZE` | No | `50` | Max events per Braintrust insert call | | `BRAINTRUST_FLUSH_INTERVAL_MS` | No | `500` | Max delay before flushing a partial batch | | `BRAINTRUST_REQUEST_TIMEOUT_MS` | No | `10000` | Per-request timeout for Braintrust insert calls | | `BRAINTRUST_MAX_RETRIES` | No | `3` | Retries for `429`, `5xx`, and timeout/connect failures | | `BRAINTRUST_RETRY_BASE_DELAY_MS` | No | `250` | Initial retry backoff | | `BRAINTRUST_RETRY_MAX_DELAY_MS` | No | `5000` | Retry backoff cap | | `BRAINTRUST_RECORD_CONTENT` | No | `false` | Export raw turn and LLM text content | | `BRAINTRUST_RECORD_THINKING` | No | `none` | Extended thinking export mode: `none`, `summary`, `full` | | `BRAINTRUST_TOOL_ARGS_MODE` | No | `redacted` | Tool argument export mode: `full`, `redacted`, `none` | | `BRAINTRUST_TOOL_RESULTS_MODE` | No | `summary` | Tool result export mode: `full`, `summary`, `redacted`, `none` | | `BRAINTRUST_DEBUG_PAYLOADS` | No | `false` | Print full outbound Braintrust payload JSON to local debug logs | ### EXTERNAL_AGENTS_ENABLED Set to `true` to enable self-service customer-hosted HTTPS Agent registration and request-driven execution. Default: disabled. Requires configured encryption and outbound HTTPS through the host egress service. The Agent’s dangerous-settings permission gates registration, rotation and disable. Customer endpoints receive no CP user or fleet credentials. See [Cloudflare agents](https://everruns.com/docs/how-to/cloudflare-agent/) for the deployment, protocol, limits and smoke test. ### CUSTOM_WORKER_GRPC_BIND_ADDR **Component:** Control plane. **Default:** unset (listener disabled). Enables the restricted customer-worker gRPC listener, separate from the trusted fleet service. Loopback development example: `127.0.0.1:28704`. Non-loopback binds require server TLS; see [Custom agent workers](https://everruns.com/docs/how-to/custom-agent-workers/) for provisioning, process-specific TLS settings, ingress and revocation. Customer binaries receive their own deployment token, never the internal fleet token. The reference data-agent worker uses `CUSTOM_WORKER_URL`, `CUSTOM_WORKER_TOKEN`, `DATA_AGENT_DATABASE` and `OPENAI_API_KEY`; HTTPS additionally uses `CUSTOM_WORKER_CA_CERT`. These are application settings, not CP secrets.
---
# Authentication Configuration Runbook
> Configure authentication modes, personal access tokens, OAuth providers, JWT secrets, token lifetimes, and sign-up controls.
Source:
## Overview This runbook covers configuring and managing authentication for Everruns. ## Authentication Modes ### 1. No Authentication (Development) Use for local development when authentication isn’t needed:
```bash
export DEPLOYMENT_GRADE=dev
export AUTH_MODE=none
```
All requests will be allowed with full admin access. `none` is refused outside the `dev` deployment grade; omitting `AUTH_MODE` does not enable unauthenticated production. If you later restart the same database with `AUTH_MODE=admin` or `AUTH_MODE=full`, startup revokes personal access tokens owned by the seeded anonymous admin user so a token minted in no-auth development cannot remain privileged after authentication is enabled. ### 2. Admin Mode (Simple Development) Use for local development with basic access control:
```bash
export AUTH_MODE=admin
export AUTH_ADMIN_EMAIL=admin@example.com
export AUTH_ADMIN_PASSWORD=your-secure-password
export AUTH_JWT_SECRET=$(openssl rand -hex 32)
```
Only the admin user can authenticate. ### 3. Full Authentication (Production) Use for production deployments:
```bash
export AUTH_MODE=full
export PUBLIC_APP_URL=https://your-domain.com
export AUTH_JWT_SECRET=$(openssl rand -hex 32)
# Optional: Configure OAuth
export AUTH_GOOGLE_CLIENT_ID=your-google-client-id
export AUTH_GOOGLE_CLIENT_SECRET=your-google-client-secret
export AUTH_GITHUB_CLIENT_ID=your-github-client-id
export AUTH_GITHUB_CLIENT_SECRET=your-github-client-secret
```
## Environment Variables Reference ### Core Settings | Variable | Required | Description | | ------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------- | | `AUTH_MODE` | Required outside `dev` grade | `none`, `admin`, `full`, or `external`. Unset resolves to `none`, which is allowed only at `dev` grade. | | `PUBLIC_APP_URL` | For OAuth | Public app origin used to derive auth callback URLs | | `AUTH_BASE_URL` | No | Override callback base URL when it differs from `PUBLIC_APP_URL` + `API_PREFIX` | | `AUTH_LOGIN_ORIGIN` | No | Trusted remote origin hosting `/login`; set identically for server and UI runtimes | | `AUTH_JWT_SECRET` | For admin/full | JWT signing secret (min 32 chars recommended) | ### Admin Mode Settings | Variable | Required | Description | | --------------------- | ---------------- | ------------------- | | `AUTH_ADMIN_EMAIL` | Yes (admin mode) | Admin user email | | `AUTH_ADMIN_PASSWORD` | Yes (admin mode) | Admin user password | ### JWT Settings | Variable | Required | Description | | --------------------------------- | -------- | ---------------------------------------------------- | | `AUTH_JWT_ACCESS_TOKEN_LIFETIME` | No | Access token lifetime in seconds (default: 900) | | `AUTH_JWT_REFRESH_TOKEN_LIFETIME` | No | Refresh token lifetime in seconds (default: 2592000) | ### Device-bound sessions Browser sign-ins support [Device Bound Session Credentials](https://w3c.github.io/webappsec-dbsc/) (DBSC) with no configuration. In a browser that implements it, currently Chrome, the sign-in response offers registration, the browser creates a key in its secure hardware, and the server stores the public key with the session’s refresh token. The browser then drops the 30-day `refresh_token` cookie and renews the short-lived `access_token` cookie itself, through `POST /v1/auth/dbsc/refresh`, with a proof signed by that key. A refresh token copied from such a browser is refused, and a copied access token works only until it expires (`AUTH_JWT_ACCESS_TOKEN_LIFETIME`). Browsers without DBSC keep the refresh cookie and `POST /v1/auth/refresh` as before. Requirements: HTTPS, and a reverse proxy that forwards `Secure-Session-Registration`, `Secure-Session-Challenge`, `Secure-Session-Response` and `Sec-Secure-Session-Id` unchanged. The registration path is relative, so serving the API under a prefix such as `/api` works. Signing out, or deleting the user’s refresh tokens (see [Force Logout User](#force-logout-user)), ends a device-bound session at its next renewal. ### Feature Toggles | Variable | Required | Description | | ----------------------- | -------- | ------------------------------------------ | | `AUTH_DISABLE_PASSWORD` | No | Set to `true` to disable password login | | `AUTH_DISABLE_SIGNUP` | No | Set to `true` to disable user registration | ### Google OAuth | Variable | Required | Description | | ----------------------------- | ---------------- | ------------------------------------- | | `AUTH_GOOGLE_CLIENT_ID` | For Google OAuth | Google OAuth client ID | | `AUTH_GOOGLE_CLIENT_SECRET` | For Google OAuth | Google OAuth client secret | | `AUTH_GOOGLE_REDIRECT_URI` | No | Custom redirect URI | | `AUTH_GOOGLE_ALLOWED_DOMAINS` | No | Comma-separated allowed email domains | ### GitHub OAuth | Variable | Required | Description | | --------------------------- | ---------------- | -------------------------- | | `AUTH_GITHUB_CLIENT_ID` | For GitHub OAuth | GitHub OAuth client ID | | `AUTH_GITHUB_CLIENT_SECRET` | For GitHub OAuth | GitHub OAuth client secret | | `AUTH_GITHUB_REDIRECT_URI` | No | Custom redirect URI | ## Common Tasks ### Generate JWT Secret
```bash
# Using OpenSSL
openssl rand -hex 32
# Using Python
python3 -c "import secrets; print(secrets.token_hex(32))"
```
### Verify Authentication is Working
```bash
# Check auth config endpoint
curl http://localhost:9300/api/v1/auth/config
# Should return:
# {"mode":"none","password_auth_enabled":false,"oauth_providers":[],"signup_enabled":false}
# For admin mode:
curl -X POST http://localhost:9300/api/v1/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"admin@example.com","password":"your-password"}'
```
### Create Personal Access Token
```bash
# Login first to get access token
TOKEN=$(curl -s -X POST http://localhost:9300/api/v1/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"user@example.com","password":"password"}' | jq -r '.access_token')
# Create personal access token
curl -X POST http://localhost:9300/api/v1/auth/personal-access-tokens \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"my-token"}'
# Read-only token: add "scopes": ["read"]
curl -X POST http://localhost:9300/api/v1/auth/personal-access-tokens \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"name":"ci-reader","scopes":["read"]}'
# Response includes the full token - save it, it's shown only once
```
Omitting `scopes` creates a full-access token (`["*"]`). A read-only token gets `403 insufficient_scope` on anything outside read access. See [Token access](https://everruns.com/docs/features/ui/#token-access). ### Revoke Personal Access Token
```bash
curl -X DELETE http://localhost:9300/api/v1/auth/personal-access-tokens/{token_id} \
-H "Authorization: Bearer $TOKEN"
```
### Force Logout User Delete their refresh tokens from database:
```sql
DELETE FROM refresh_tokens WHERE user_id = 'user-uuid-here';
```
## Troubleshooting ### “Authentication required” when AUTH_MODE=none * Verify `AUTH_MODE` environment variable is set correctly * Restart the server after changing environment variables ### JWT Validation Fails * Ensure `AUTH_JWT_SECRET` hasn’t changed * Check token hasn’t expired * Verify the token is for the correct environment ### OAuth Redirect Fails * Verify `AUTH_BASE_URL` matches the OAuth app configuration, or that `PUBLIC_APP_URL` derives the expected `{PUBLIC_APP_URL}/api` base * Check that redirect URI in provider matches `{AUTH_BASE_URL}/v1/auth/callback/{provider}` or `{PUBLIC_APP_URL}/api/v1/auth/callback/{provider}` * If you set `AUTH_BASE_URL`, ensure it already includes your REST API prefix (default: `/api`) * Ensure client ID and secret are correct ### Password Login Returns Unauthorized * In admin mode: check `AUTH_ADMIN_EMAIL` and `AUTH_ADMIN_PASSWORD` match * In full mode with password disabled: check `AUTH_DISABLE_PASSWORD` isn’t set * Verify user exists and password is correct ## Database Migration Authentication tables are included in the base schema and **auto-applied on server startup**. No manual migration step is required. To check migration status:
```bash
# Via admin container
docker run --rm -e DATABASE_URL="$DATABASE_URL" everruns-admin migrate-info
```
## Health Check The `/health` endpoint shows current auth mode:
```bash
curl http://localhost:9300/health
# {"status":"ok","version":"0.2.0","auth_mode":"None"}
```
## Agent Discovery Endpoints The server publishes public documents so an AI agent that only knows the origin can find the MCP endpoint and work out how to authenticate before it has any credentials: | Path | Contents | | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `/llms.txt` | What the server offers and where to start: MCP endpoint, auth.md, REST API, OpenAPI, docs | | `/auth.md` | How to obtain credentials: OAuth 2.1 with dynamic client registration for the MCP endpoint, and personal access tokens for the REST API | | `/.well-known/mcp/server-card.json` | MCP Server Card (SEP-1649): server identity, transport, capabilities, and where its OAuth metadata lives | | `/.well-known/api-catalog` | API catalog (RFC 9727): the REST API with its OpenAPI document, and the MCP endpoint | | `/.well-known/oauth-protected-resource` | Root copy of the MCP endpoint’s protected resource metadata (RFC 9728), for clients that probe the bare path | | `/robots.txt`, `/sitemap.xml` | Keep crawlers out of the console, allow the documents above, declare Content Signals | All are generated from the running configuration rather than hardcoded, so a self-hosted deployment describes itself: * URLs come from `AUTH_BASE_URL` / `BASE_URL` (or `PUBLIC_APP_URL` plus `API_PREFIX`), the same value used for OAuth callbacks and the MCP resource binding. Set them to the public origin, not an internal container address, or the documents will advertise URLs an agent cannot reach. * Content follows `AUTH_MODE`. Under `AUTH_MODE=none` the documents state that no credentials are required instead of describing an OAuth flow that is not enforced. * The server card `version` and `serverInfo.version` are the running server’s release version, the same value `/health` reports. A release updates them with the binary. ### Reverse proxy configuration (required) `/auth.md`, `/llms.txt`, `/robots.txt`, and `/sitemap.xml` sit at the server root, so a deployment that puts the UI on `/` must route these paths to the server explicitly. Without the rule the request falls through to the UI and returns its 404, and the document is unreachable even though the server serves it. The proxy also owns two behaviours of `/` itself: a `Link` header pointing at the discovery documents, and `Accept: text/markdown` answered with `/llms.txt`, since the console renders nothing without JavaScript. The bundled [local proxy](https://github.com/everruns/everruns/blob/main/local/Caddyfile), [Railway proxy](https://github.com/everruns/everruns/blob/main/infra/railway/caddy/Caddyfile), and [Docker Compose stack](https://github.com/everruns/everruns/blob/main/infra/docker-compose/docker-compose.yaml) already include it. For a custom proxy, add these rules next to wherever `/.well-known/*` is routed:
```caddyfile
handle /.well-known/* {
reverse_proxy server:9000
}
@agent_docs path /auth.md /llms.txt /robots.txt /sitemap.xml
handle @agent_docs {
reverse_proxy server:9000
}
@agent_markdown {
path /
header Accept *text/markdown*
}
handle @agent_markdown {
header Vary Accept
rewrite * /llms.txt
reverse_proxy server:9000
}
handle {
header / +Link "; rel=\"api-catalog\", ; rel=\"describedby\"; type=\"text/markdown\""
reverse_proxy ui:9305
}
```
nginx (documents only):
```nginx
location ~ ^/(auth\.md|llms\.txt|robots\.txt|sitemap\.xml)$ {
proxy_pass http://server:9000;
}
```
Verify after deploying:
```bash
curl -fsS https://your-host/auth.md | head -1 # expect "# auth.md"
curl -fsS -H 'Accept: text/markdown' https://your-host/ | head -1 # expect "# Everruns"
curl -fsSI https://your-host/ | grep -i '^link' # expect rel="api-catalog"
curl -fsS https://your-host/.well-known/mcp/server-card.json
```
A response of `text/html` rather than `text/markdown` means the UI answered and the proxy rule is missing. ## Security Best Practices 1. **Never commit secrets**: Use environment variables or secret management 2. **Rotate JWT secret**: Change `AUTH_JWT_SECRET` periodically (invalidates all tokens) 3. **Use HTTPS**: Always use HTTPS in production for OAuth callbacks 4. **Limit OAuth domains**: Use `AUTH_GOOGLE_ALLOWED_DOMAINS` to restrict access 5. **Monitor personal access token usage**: Track `last_used_at` for suspicious activity 6. **Set token expiration**: Use shorter `AUTH_JWT_ACCESS_TOKEN_LIFETIME` for higher security
---
# Durable Execution Engine Setup
> Run Everruns with the PostgreSQL-backed durable execution engine: database setup, migrations, and worker configuration.
Source:
This guide explains how to run Everruns with the custom PostgreSQL-backed durable execution engine. ## Overview Durable execution is PostgreSQL-backed and provides: * Turns that survive worker crashes: each session is leased by one worker, and the turn resumes from the session’s event log on another * Background tasks with automatic retries * Distributed task queue with backpressure support * Circuit breakers and dead letter queues * No additional infrastructure required (uses existing PostgreSQL) ## Quick Start ### 1. Prerequisites * PostgreSQL running and accessible * `DATABASE_URL` environment variable set * Server startup applies the embedded migrations, including session actors and background-task tables ### 2. Start the API The Platform uses session actors for agent turns and the durable task queue for background work; there is no legacy workflow mode to select. For a local-only two-process setup, configure a development deployment grade without enabling `DEV_MODE` (which runs workers in-process):
```bash
export DATABASE_URL="postgres://postgres:postgres@localhost/everruns"
export DEPLOYMENT_GRADE=dev
export AUTH_MODE=none
export WORKER_GRPC_AUTH_TOKEN="$(openssl rand -hex 32)"
export SECRETS_ENCRYPTION_KEY="kek-v1:$(openssl rand -base64 32)"
# Start the API server
cargo run -p everruns-server
```
### 3. Start Durable Worker In a separate terminal:
```bash
# Workers need the matching gRPC token, but NO DATABASE_URL.
export WORKER_GRPC_AUTH_TOKEN=""
export SERVER_GRPC_ADDRESS="127.0.0.1:9001"
# Start the task worker
cargo run -p everruns-worker
```
For a non-development deployment, configure `AUTH_MODE=admin`, `full`, or `external` and its required settings as described in [Authentication](https://everruns.com/docs/sre/runbooks/authentication/). Keep the encryption key stable across restarts. **Important:** Workers communicate with the control-plane via gRPC and do not require direct database access. This improves security and simplifies deployment. Or programmatically:
```rust
use everruns_worker::{TaskWorkerConfig, WorkerAppBuilder};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
WorkerAppBuilder::new(TaskWorkerConfig::from_env())
.run()
.await
}
```
## Configuration ### Environment Variables | Variable | Description | Default | | -------------------------- | ------------------------------------------------------------------- | ------------------------------- | | `DATABASE_URL` | PostgreSQL connection URL (server only; workers do not need it) | Required | | `SERVER_GRPC_ADDRESS` | Server gRPC address (`WORKER_GRPC_ADDRESS` legacy alias) | `127.0.0.1:9001` | | `WORKER_GRPC_AUTH_TOKEN` | Same bearer token on server and workers | Required unless `DEV_MODE=true` | | `WORKER_ID` | Unique worker identifier | Auto-generated | | `MAX_CONCURRENT_TASKS` | Max queued background tasks per worker (session turns do not count) | `50` | | `WORKER_MAX_RUNNING_TURNS` | Max session turns per worker (`0` = from the memory limit) | from the memory limit | ### Database Tables Current embedded migrations create two distinct sets of records. Agent turns use: * `session_actors` - Runnable state, lease ownership, fencing and inbox cursor * `events` - Durable session journal used to reconstruct turns The generic background-task subsystem also retains: * `durable_workflow_instances` - Workflow state and metadata * `durable_workflow_events` - Event sourcing log * `durable_task_queue` - Distributed task queue * `durable_dead_letter_queue` - Failed tasks for manual inspection * `durable_workers` - Worker registration and heartbeats * `durable_circuit_breaker_state` - Circuit breaker states ## Turn Lifecycle Turns do not run on the task queue. Each session is an actor in `session_actors`: 1. **Message Created**: the API stores the message and marks the session runnable in the same transaction 2. **Lease**: a worker leases the session (`LeaseActors`) and runs the whole turn, renewing the lease every few seconds 3. **Steering**: messages that arrive mid-turn reach the turn before its next model call 4. **Completion**: the worker releases the session; a message that arrived as the turn ended keeps it runnable for the next turn 5. **Crash**: an unrenewed lease expires after 10 seconds and another worker resumes the turn from the session’s event log The task queue carries the background activities: leased-resource cleanup, the session task reaper, and scheduled channel and agent-trigger invocations. ### Check Running Sessions
```sql
SELECT session_id, org_id, lease_holder, lease_expires_at, runnable_since, attempts
FROM session_actors
WHERE runnable_since IS NOT NULL OR lease_holder IS NOT NULL
ORDER BY runnable_since;
```
A session runnable for a long time with no holder means no worker is leasing (workers down or drained). A high `attempts` means its turn keeps losing its worker; past `DURABLE_NO_PROGRESS_SEAL_THRESHOLD` (default 3) the next worker seals the turn. ## Monitoring ### Check Generic Workflow Records These are not session turns. Use the `session_actors` query above to inspect running agents.
```sql
SELECT id, workflow_type, status, created_at
FROM durable_workflow_instances
WHERE status IN ('pending', 'running')
ORDER BY created_at DESC;
```
### Check Pending Tasks
```sql
SELECT id, workflow_id, activity_type, status, attempt
FROM durable_task_queue
WHERE status = 'pending'
ORDER BY created_at;
```
### Check Dead Letter Queue
```sql
SELECT id, workflow_id, activity_type, last_error, dead_at
FROM durable_dead_letter_queue
ORDER BY dead_at DESC;
```
### Check Worker Status
```sql
SELECT id, status, current_load, last_heartbeat_at
FROM durable_workers
WHERE status = 'active';
```
## Crash Recovery Session turns recover through lease expiry and journal reconstruction described above. The heartbeat and stale-task mechanisms below apply to queued background activities, not to each reason/tool step of an agent turn. ### Worker Heartbeats Workers send heartbeats every 10 seconds while executing tasks. If a worker crashes: 1. The task remains in `claimed` status with stale `heartbeat_at` 2. Control-plane background task detects stale tasks (30s threshold) 3. Stale tasks are automatically reset to `pending` status 4. Another worker can claim and retry the task ### Stale Task Reclamation The control-plane runs a background task (every 10s) that: * Finds tasks with `status = 'claimed'` and `heartbeat_at` older than 30s * Resets them to `pending` status * Logs reclaimed task IDs for monitoring
```sql
-- View tasks that may need reclamation
SELECT id, workflow_id, activity_type, claimed_by, heartbeat_at
FROM durable_task_queue
WHERE status = 'claimed'
AND heartbeat_at < NOW() - INTERVAL '30 seconds';
```
## Troubleshooting ### Worker Not Processing Tasks 1. Check worker is running and connected to correct `SERVER_GRPC_ADDRESS` 2. Verify `activity_types` match task types in queue 3. Check worker heartbeat in `durable_workers` table ### Background Tasks Stuck in Running 1. Check for claimed tasks that haven’t completed 2. Look for errors in worker logs 3. Check DLQ for failed tasks 4. Wait for stale task reclamation (30s threshold) ### Task Retries Exhausted Tasks moved to DLQ after exhausting retries:
```sql
-- View DLQ entries
SELECT * FROM durable_dead_letter_queue ORDER BY dead_at DESC;
```
To retry after addressing the cause, use the platform-admin endpoint `POST /v1/durable/dlq/{dlq_id}/retry` (under your configured API prefix). It creates a new queued task and records the requeue in one transaction. Updating `requeued_at` alone does not enqueue any work. Inspect the task’s external effects before retrying it; a retry can repeat them. Metrics and tracing are covered under [Observability](https://everruns.com/docs/observability/opentelemetry/).
---
# Encryption Key Rotation
> Rotate the secrets encryption key (KEK): key deployment, data re-encryption, and old key removal.
Source:
This runbook describes how to rotate the secrets encryption key (KEK) used to encrypt sensitive data in the database. ## Overview Everruns uses envelope encryption with versioned keys. Key rotation is a multi-phase process: 1. **Deploy new key** alongside old key 2. **Re-encrypt data** from old key to new key 3. **Remove old key** after all data is migrated ## Prerequisites * Access to secrets management (environment config or secrets manager) * Ability to run the admin container in production * Ability to deploy application updates * The `reencrypt-secrets` CLI tool (available in the admin container) ## Rotation Procedure ### Phase 1: Generate New Key Generate a new encryption key with an incremented version:
```bash
# Generate new key (increment version number from current)
python3 -c "import os, base64; print('kek-v2:' + base64.b64encode(os.urandom(32)).decode())"
```
Store the output securely. Example output:
```plaintext
kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w=
```
### Phase 2: Deploy with Both Keys Update environment configuration:
```bash
# Current key becomes the new one
SECRETS_ENCRYPTION_KEY=kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w=
# Previous key is preserved for decryption
SECRETS_ENCRYPTION_KEY_PREVIOUS=kek-v1:8B3uCQ4Znx45hl5nB+PKVriRrj/KtEVM+wBZ2VGa9vY=
```
Deploy the application with both keys configured. At this point: * **New encryptions** use `kek-v2` * **Existing data** encrypted with `kek-v1` is still decryptable ### Phase 3: Re-encrypt Existing Data Use the `reencrypt-secrets` CLI tool to migrate all data to the new key. #### Step 1: Dry Run (Preview Changes) First, run in dry-run mode to see what would be re-encrypted:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --dry-run
```
Example output:
```plaintext
2024-01-15T10:00:00Z INFO Encryption service initialized. Primary key: kek-v2
2024-01-15T10:00:00Z INFO Available keys: ["kek-v2", "kek-v1"]
2024-01-15T10:00:00Z INFO Connected to database
2024-01-15T10:00:00Z INFO Processing table: llm_providers
2024-01-15T10:00:01Z INFO Would re-encrypt llm_providers.api_key_encrypted (id=..., current_key=kek-v1)
2024-01-15T10:00:01Z INFO DRY RUN: Would re-encrypt 42 of 100 records
```
#### Step 2: Execute Re-encryption Once satisfied with the dry run, execute the actual re-encryption:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --batch-size 50
```
#### CLI Options
```plaintext
USAGE:
reencrypt-secrets [OPTIONS]
OPTIONS:
-n, --dry-run Show what would be changed without making changes
-b, --batch-size Process N records at a time (default: 100)
-t, --table Only process specified table (default: all)
-h, --help Show this help message
```
### Phase 4: Verify Migration Confirm all data has been migrated by running another dry run:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt --dry-run
```
Expected output:
```plaintext
2024-01-15T11:00:00Z INFO DRY RUN: Would re-encrypt 0 of 100 records
```
You can also verify directly in the database:
```sql
-- Check for any remaining records with old key
SELECT COUNT(*)
FROM llm_providers
WHERE api_key_encrypted::text LIKE '%"key_id":"kek-v1"%';
-- Should return 0
```
### Phase 5: Remove Old Key Once verified, remove the old key from configuration:
```bash
# Remove the previous key
SECRETS_ENCRYPTION_KEY=kek-v2:xR7qW2mN9pL4kJ8vB3tY6fE1hG5sD0cA9uI7oP2nM6w=
# SECRETS_ENCRYPTION_KEY_PREVIOUS= (remove or leave empty)
```
Deploy the updated configuration. **Important**: Keep the old key archived securely for disaster recovery. You may need it if backup restoration is required. ## Rollback Procedure If issues occur during rotation: ### During Phase 2-3 (Both Keys Active) No rollback needed - both keys work. Simply stop the re-encryption CLI if causing issues. ### After Phase 5 (Old Key Removed) If old key was removed but some data wasn’t migrated: 1. Re-add the old key as `SECRETS_ENCRYPTION_KEY_PREVIOUS` 2. Deploy 3. Run the re-encryption CLI again 4. Verify again before removing ## Monitoring During rotation, monitor: * **CLI Progress Output**: The tool logs progress every 1000 records * **API Error Rates**: Watch for decryption failures in application logs * **Database Load**: Ensure re-encryption isn’t causing performance issues ## Emergency: Compromised Key If a key is suspected compromised: 1. **Immediately** generate new key and deploy with both keys 2. Run re-encryption CLI with **highest priority**:
```bash
docker run --rm \
-e DATABASE_URL="$DATABASE_URL" \
-e SECRETS_ENCRYPTION_KEY="kek-v2:..." \
-e SECRETS_ENCRYPTION_KEY_PREVIOUS="kek-v1:..." \
everruns-admin reencrypt
```
3. Remove compromised key as soon as all data is migrated 4. Rotate any credentials that may have been exposed ## Key Storage Best Practices * Store keys in a secrets manager (AWS Secrets Manager, HashiCorp Vault, etc.) * Enable audit logging for key access * Rotate keys on a regular schedule (e.g., annually) * Keep previous key archived for disaster recovery (separate secure storage) * Never commit keys to source control
---
# Migration 144 Fails on Duplicate Platform Chat Starters
> Clear user-set platform-chat-starter tags that block migration 144's unique index, then restart.
Source:
Migration `144_platform_chat_starter_unique.sql` can fail on startup with a duplicate key violation, which stops the server before it finishes migrating.
```plaintext
ERROR: could not create unique index "idx_sessions_platform_chat_starter_owner"
DETAIL: Key (org_id, owner_principal_id)=(…, …) is duplicated.
```
## Why it happens Migration 144 makes `platform-chat-starter` the marker for an owner’s single automatic Platform Chat, and enforces it with a partial unique index on `(org_id, owner_principal_id)`. Before 144 that tag was not reserved, and `sessions.tags` is user-editable. Any session could already carry it. The migration elects a canonical starter only among sessions that do **not** already have the tag:
```sql
AND NOT 'platform-chat-starter' = ANY(s.tags);
```
So two pre-existing, user-tagged sessions belonging to one owner are both skipped by the election and then collide on the unique index. ## Why this needs an operator The fix cannot ship as a migration: * Migration 144 is immutable once merged. SQLx records each migration’s checksum in `_sqlx_migrations`, so editing it breaks startup on every database that already applied the original. * A later migration cannot help either. SQLx applies in version order and halts on failure, so a database stuck at 144 never reaches 145 or beyond. A database in this state has to be cleared by hand before the server can start. ## Diagnosis Find the owners with more than one tagged session:
```sql
SELECT org_id, owner_principal_id, count(*) AS tagged, array_agg(id) AS session_ids
FROM sessions
WHERE 'platform-chat-starter' = ANY(tags)
GROUP BY org_id, owner_principal_id
HAVING count(*) > 1;
```
No rows means this is not the cause and the failure is something else. ## Remediation Every pre-144 use of the tag is user-supplied and carries no meaning the product assigns to it, so clear all of them and let the migration elect starters itself:
```sql
BEGIN;
-- Confirm the blast radius before changing anything.
SELECT count(*) FROM sessions WHERE 'platform-chat-starter' = ANY(tags);
UPDATE sessions
SET tags = array_remove(tags, 'platform-chat-starter')
WHERE 'platform-chat-starter' = ANY(tags);
COMMIT;
```
Then restart the server. Migration 144 runs from a clean state: it elects one starter per owner, creates the unique index, and the triggers keep the tag authoritative from then on. ## Verification
```sql
-- migration recorded
SELECT version, description, success FROM _sqlx_migrations WHERE version = 144;
-- at most one starter per owner
SELECT org_id, owner_principal_id, count(*)
FROM sessions
WHERE 'platform-chat-starter' = ANY(tags)
GROUP BY org_id, owner_principal_id
HAVING count(*) > 1;
```
The first should report `success = true`; the second should return no rows. ## Notes Running the `UPDATE` on a database where 144 already succeeded is not useful and is best avoided: the tag is authoritative there, the `keep_platform_chat_starter_tag` trigger restores it on update, and the unique index already guarantees there is nothing to clean.
---
# Virtual User Cutover
> Upgrade runtime accounts and connections without duplicating credentials across organizations.
Source:
This upgrade replaces agent identities and the global user connection store with organization-scoped virtual users. Existing identity IDs, session IDs, conversation history, archive state, and pins are preserved. Everruns login accounts, roles, organization membership, and personal access tokens remain management identities. Stop old API and worker processes before running this upgrade. The schema rename and internal protocol change require all processes to use the new version together; mixed-version rolling deployment is unsupported. Drain active runs before stopping workers. Existing historical messages are retained, while new execution records capture the current speaker and responder. Restart the upgraded API to apply migrations, then start upgraded workers and UI. Accounts with one organization have their encrypted provider grants moved automatically. Accounts with several organizations retain grants in a pending cutover queue that execution cannot read. Each account owner chooses a single destination in Settings → Connections. The move preserves the grant ID and encrypted payload, and rekeys resource cleanup to that virtual user. An existing destination grant is never overwritten. Resources awaiting an ambiguous grant move cannot be cleaned up until a grant is available on their organization-scoped owner. Moving a grant unblocks resources in the chosen organization. Resources in other organizations require explicit reconnection there; credentials are never copied between organizations. Legacy session-local OAuth grants have no verified caller binding and require reconnection. Agents continue using their existing service accounts. External identities retain qualified workspace bindings where available; unqualified historical actors require verified ingress before becoming a new runtime subject. After restart, check health, open an existing chat, and verify its pin and archive state. Compare provider grants on Settings → Connections and the default virtual user’s Connections tab. Test a current user turn, a service turn, and an unattended user attachment: the last must request setup rather than borrow a human owner’s grant. Confirm the pending queue is empty only after users have selected destinations. Keep backups and the existing encryption keys throughout the upgrade.
---
# System Doctor
> Check that a self-hosted Everruns install is set up right and working, and see what to fix when it is not
Source:
The system doctor checks every part of an install and reports each finding in plain words: what it found, why it matters and how to fix it. It covers the database and its migrations, connected workers (their load, versions, and what each worker finds about its own setup, such as whether it reaches the egress proxy), every running API server and whether they all run the same version, sign-in and encryption keys, NATS, Valkey, S3 file storage and monitoring. It works the same way however you host Everruns. It reads your configuration, so a part you do not use (for example NATS on a single server) shows as a note explaining what that means, not as a failure. On a production install (`DEPLOYMENT_GRADE=prod`, the default) unsafe settings such as sign-in turned off are problems; on a development install they are warnings. Each API server records itself every 30 seconds, so the report is the same whichever server answers it. Once a second server runs, NATS and Valkey stop being optional notes: without NATS each server polls the database for the others’ updates, and without Valkey each server allows the full rate limit on its own. Servers on different versions are a warning while a deploy rolls out and a problem after ten minutes. ## Run it In the app, open **System → Health**. It shows the verdict, what needs attention with how to fix it, and every other check by area. **Check again** runs the checks now; **Copy report** copies a plain-text report to paste into an issue. The page and the API need a platform user. From the CLI:
```bash
everruns system doctor --refresh true
```
The same report is at `GET /v1/system/doctor`. Without `refresh` you get the report from the last minute; the checks write a small test file to S3 and connect to NATS and Valkey, so they do not run on every request. Each check has a `status`: | Status | Meaning | | --------- | ---------------------------------------------------------- | | `ok` | Working as intended. | | `info` | Not used by this install, or worth knowing. Never a fault. | | `warning` | Works, but degraded or risky. | | `problem` | Broken or unsafe now. | The report’s own `status` is the worst of its checks, ignoring `info`, so a script can fail on `problem`:
```bash
everruns system doctor --refresh true --output json | jq -e '.status != "problem"'
```
## Without signing in The server binary runs the doctor on its own, against the services its environment names, and exits `1` when a check is a problem (warnings exit `0`). It needs no running server and no sign-in, so it works as a check before a deploy and from inside a container:
```bash
everruns-server --doctor # plain text
everruns-server --doctor --json # the API's report shape
docker compose exec server /app/everruns-server --doctor
```
`everruns docker doctor` runs it for you when your local instance is up. ## Alerting Every server exports each check as a Prometheus gauge, `everruns_doctor_check_status{area,check}`, refreshed every minute: `0` ok or not used, `1` warning, `2` problem. `everruns_doctor_status` is the whole report on the same scale. Every replica exports the same checks, so alert on the cluster’s worst value:
```text
max without(instance) (everruns_doctor_check_status) == 2
```
## Adding your own checks If you build your own server on `everruns-server`, report your own services in the same page, gauges and exit code. Implement `DoctorExtension` and register it on your host composition:
```rust
use everruns_server::domains::system::doctor::{
CheckArea, DoctorCheck, DoctorExtension, DoctorExtensions, DoctorFindings,
};
struct Billing;
#[async_trait::async_trait]
impl DoctorExtension for Billing {
fn name(&self) -> &str {
"billing"
}
async fn run(&self) -> DoctorFindings {
let mut findings = DoctorFindings::default();
let check = DoctorCheck::new(CheckArea::Integrations, "integrations.billing", "Billing");
findings.check(if std::env::var("BILLING_KEY").is_ok() {
check.ok("Billing is configured.")
} else {
check.warning("BILLING_KEY is not set.").fix("Set BILLING_KEY.")
});
findings
}
}
composition.insert_extension(Arc::new(DoctorExtensions::new(vec![Arc::new(Billing)])));
```
Your checks show under **Integrations**. Report whether a setting is present, never its value, and keep network calls short: an extension that does not finish within 8 seconds shows as a warning. ## Readiness probe `GET /health/ready` answers `200` once the database is reachable and its schema matches the server’s version, and `503` otherwise. It needs no sign-in. Use it as the readiness probe, and keep `/health` as the liveness probe so a database outage does not get healthy servers restarted.