System Doctor
Check that a self-hosted Everruns install is set up right and working, and see what to fix when it is not
The system doctor checks every part of an install and reports each finding in plain words: what it found, why it matters and how to fix it. It covers the database and its migrations, connected workers (their load, versions, and what each worker finds about its own setup, such as whether it reaches the egress proxy), every running API server and whether they all run the same version, sign-in and encryption keys, NATS, Valkey, S3 file storage and monitoring.
It works the same way however you host Everruns. It reads your configuration,
so a part you do not use (for example NATS on a single server) shows as a note
explaining what that means, not as a failure. On a production install
(DEPLOYMENT_GRADE=prod, the default) unsafe settings such as sign-in turned
off are problems; on a development install they are warnings.
Each API server records itself every 30 seconds, so the report is the same whichever server answers it. Once a second server runs, NATS and Valkey stop being optional notes: without NATS each server polls the database for the others’ updates, and without Valkey each server allows the full rate limit on its own. Servers on different versions are a warning while a deploy rolls out and a problem after ten minutes.
Run it
Section titled “Run it”In the app, open System → Health. It shows the verdict, what needs attention with how to fix it, and every other check by area. Check again runs the checks now; Copy report copies a plain-text report to paste into an issue. The page and the API need a platform user.
From the CLI:
everruns system doctor --refresh trueThe same report is at GET /v1/system/doctor. Without refresh you get the
report from the last minute; the checks write a small test file to S3 and
connect to NATS and Valkey, so they do not run on every request.
Each check has a status:
| Status | Meaning |
|---|---|
ok |
Working as intended. |
info |
Not used by this install, or worth knowing. Never a fault. |
warning |
Works, but degraded or risky. |
problem |
Broken or unsafe now. |
The report’s own status is the worst of its checks, ignoring info, so a
script can fail on problem:
everruns system doctor --refresh true --output json | jq -e '.status != "problem"'Without signing in
Section titled “Without signing in”The server binary runs the doctor on its own, against the services its
environment names, and exits 1 when a check is a problem (warnings exit
0). It needs no running server and no sign-in, so it works as a check before
a deploy and from inside a container:
everruns-server --doctor # plain texteverruns-server --doctor --json # the API's report shapedocker compose exec server /app/everruns-server --doctoreverruns docker doctor runs it for you when your local instance is up.
Alerting
Section titled “Alerting”Every server exports each check as a Prometheus gauge,
everruns_doctor_check_status{area,check}, refreshed every minute: 0 ok or
not used, 1 warning, 2 problem. everruns_doctor_status is the whole
report on the same scale. Every replica exports the same checks, so alert on
the cluster’s worst value:
max without(instance) (everruns_doctor_check_status) == 2Adding your own checks
Section titled “Adding your own checks”If you build your own server on everruns-server, report your own services in
the same page, gauges and exit code. Implement DoctorExtension and register
it on your host composition:
use everruns_server::domains::system::doctor::{ CheckArea, DoctorCheck, DoctorExtension, DoctorExtensions, DoctorFindings,};
struct Billing;
#[async_trait::async_trait]impl DoctorExtension for Billing { fn name(&self) -> &str { "billing" }
async fn run(&self) -> DoctorFindings { let mut findings = DoctorFindings::default(); let check = DoctorCheck::new(CheckArea::Integrations, "integrations.billing", "Billing"); findings.check(if std::env::var("BILLING_KEY").is_ok() { check.ok("Billing is configured.") } else { check.warning("BILLING_KEY is not set.").fix("Set BILLING_KEY.") }); findings }}
composition.insert_extension(Arc::new(DoctorExtensions::new(vec![Arc::new(Billing)])));Your checks show under Integrations. Report whether a setting is present, never its value, and keep network calls short: an extension that does not finish within 8 seconds shows as a warning.
Readiness probe
Section titled “Readiness probe”GET /health/ready answers 200 once the database is reachable and its
schema matches the server’s version, and 503 otherwise. It needs no sign-in.
Use it as the readiness probe, and keep /health as the liveness probe so a
database outage does not get healthy servers restarted.