🜨 ORRERY Analysis Backend

Every test run, in one queryable record.

Orrery is Cthonica’s analysis backend. Astrolabe sends it every run from the bench. Orrery checks each one against rules you write, explains what changed, and keeps a record your QA, firmware, and release teams can search months later.

Follow one run Read the API
Channel 3 temperature for run-20251014-140311 The trace stays inside the two-sigma band around the golden run until about 790 seconds, then climbs out of it. At 812 seconds it reads 4.2 degrees above baseline and the drift rule fires. 64 68 72 76 °C 0 300 600 900 s golden run +2σ t = 812 s

Channel 3 temperature during soak, from run-20251014-140311 on DEV-00391, firmware v2.3.1.

Shaded band
Two standard deviations either side of the golden run for plan-thermal-v4.
At 812 seconds
Channel 3 reads 74.1 °C, 4.2 °C over baseline. The drift rule fires.
On the same run
Stall recovery takes 312 ms against a 180 ms baseline. Orrery links the two.

What happens to that run

The bench finished at 14:18 on 14 October. Here is everything Orrery did with the run, in order, with the files it read and wrote along the way.

  1. Astrolabe uploads the run

    Astrolabe asks for presigned URLs, then streams the manifest and NDJSON artifacts straight to storage. Uploads are idempotent by run_id and resume where they stopped, so a dropped bench connection never makes a duplicate. 

    POST /api/v1/runs/upload
{
  "run_id": "run-20251014-140311",
  "project": "prod-validation",
  "plan": "plan-thermal-v4",
  "manifest_sha256": "a7b3…"
}

  2. Checksums and schema are verified

    Every file is checked against its SHA-256 from the bench. The manifest is checked against the schema before anything else reads it, so a malformed run is rejected at the door instead of surfacing later as a strange chart.

    Verification results
    FileSizeResult
    manifest.json4.1 KBsha256 matches
    channels.ndjson10,840 rowssha256 matches
    events.ndjson212 rowssha256 matches
    schemav3valid

  3. A rule fires

    Rules are YAML files versioned with your test plan: thresholds, regressions against a golden run, schema checks, and signal shape. Each rule is pinned to the plan version that ran it, so re-checking an old run gives the same answer it gave then. 

    # rules/thermal/temp_drift.yaml
id:        temp_drift_exceeds_2σ
kind:      regression
severity:  critical
plan:      plan-thermal-v4
match:
  channel:  "channels.ndjson:temp_c"
  where:    "phase == 'soak'"
assert:
  type:       band
  baseline:   plan.golden_run
  tolerance:  2    # σ
  duration_ms: 500

  4. Claude writes the analysis

    Claude Sonnet reads the run’s artifacts next to the golden run and writes down what changed, which rules fired, and the likely cause. Every claim points at the rows it came from, so an engineer can check it in one click. 

    {
  "verdict": "failed",
  "rules_fired": ["temp_drift_exceeds_2σ",
                  "stall_recovery_slow"],
  "likely_cause": "thermal protection is throttling
    the stall recovery loop",
  "evidence": [
    { "artifact": "channels.ndjson",
      "lines": "8144-8210", "value": 74.1 },
    { "artifact": "channels.ndjson",
      "line": 9002, "stall_recovery_ms": 312 }
  ],
  "compared_to": "run-20251013-174501"
}

  5. CI hears about it

    The run.analyzed webhook fires as soon as the analysis is saved. CI, Slack, or a release script can act on the verdict without polling. 

    {
  "event": "run.analyzed",
  "run_id": "run-20251014-140311",
  "verdict": "failed",
  "permalink": "https://orrery…/r/…140311"
}

  6. The device’s history grows by one

    Every serial number has a record: firmware history, calibration events, and every run it has produced. DEV-00391’s lifetime pass rate now includes this failure.

    Device
    DEV-00391
    Runs
    238
    Lifetime pass rate
    94.1%
    Firmware
    v2.2.8, v2.2.9, v2.3.1
    Last calibrated
    2025-09-30

  7. Someone finds it again

    Two days later a firmware engineer asks which v2.3 runs tripped the drift rule. Filter by project, device, firmware, status, rule, plan version, or any manifest field, then share the URL or export the results as CSV or JSONL. 

    GET /api/v1/runs?project=prod-validation
    &fw=v2.3.*
    &rule=temp_drift_exceeds_2σ
    &status=failed

# 3 runs, newest first

  8. The release report is signed

    Sign-off produces a PDF and JSON record that can’t be edited. It links to the exact manifests, artifacts, and rules behind it, including this failure and its fix. Re-running the analysis later will not change the signed record.

    Report
    release-2.3.1
    Runs included
    214
    Failures, each linked
    3
    Signed by
    QA lead
    pdf_sha256
    9c1e…f04a

What your team opens each morning

One project: the runs that landed, the rules that fired, and links to each analysis. Open a run or filter the list. The line above the tabs is the link you would share, and it uses the same filters as the API.

Link to this view /dashboard?project=prod-validation

prod-validation

Recent runs

RunDeviceFirmwarePlanStatus

Rules fired today

  • Critical DEV-00391, channel 3, +4.2 °C
  • Warning DEV-00391, 312 ms against 180 ms
  • Info DEV-00847, v2.3.1 first seen at 14:30

Pick a rule to see every recent run it fired on.

What Orrery stores

Seven entities. Every screen, API call, and report is a view over them.

Run
id project device_id firmware plan_id status started_at finished_at manifest_sha256
Manifest
run_id env inputs capture_source astrolabe_version schema_version
Artifact
run_id kind s3_uri sha256 bytes rows
Device
id serial model firmware_history first_seen runs_total
Plan
id version rules golden_run channels
Rule
id kind severity spec plan_version
Report
id scope runs signed_by signed_at pdf_sha256

Three endpoints carry the whole product

Astrolabe calls these so you don’t have to. The dashboard, rules engine, and reports are all built on the same three.

MethodPath or eventWhat it does
POST /api/v1/runs/upload Returns presigned URLs for the manifest and artifacts. Idempotent on run_id.
GET /api/v1/runs Queries runs by any manifest field, with cursor pagination. Uses the same filters as dashboard URLs.
Webhook run.analyzed Sent when an analysis is saved, with the verdict, rules fired, and a permalink.

Before and after a shared backend

Shared drives and tribal knowledge

  • Each engineer has their own folder layout, and the schema lives in someone’s head.
  • A failed run is a screenshot in Slack and a CSV nobody can find a week later.
  • Regressions are caught when a customer files a ticket, not when the bench produces them.
  • Release sign-off is a meeting, and the evidence is whatever was on screen at the time.

One backend, one schema, one URL per run

  • Every run arrives with its manifest, checksums, and plan version, and the schema is enforced on upload.
  • A failed run has a permalink, its artifacts, the rules that fired, and a written analysis.
  • Regressions show up when the rule fires, and the webhook tells CI before anyone opens the dashboard.
  • Release sign-off is a signed report that traces back to the exact files behind it.

Tour Orrery with your own data

A 30-minute walkthrough with your test data, your devices, and one of your plans set up. No slides.

Book a tour Start with Astrolabe

Hosted, or single-tenant on request. Analysis by Claude Sonnet. Runs stored in S3 and Postgres.