Orrery · Hardware QA backend

Every test run, in one queryable record.

Orrery is the analysis backend for Cthonica. It receives runs from Astrolabe, validates them against rules you define, and turns months of bench output into a project-scoped record your QA, engineering, and release teams can interrogate — not a folder full of CSVs nobody owns.

See it in action Read the API

SaaS · single-tenant available Claude Sonnet for analysis S3 + Postgres storage

orrery.cthonica.io / dashboard prod-validation

Pass rate · 7d

92%

+3.1

Failed · 24h

3

+2

Median runtime

14:22

Reports · 7d

58

done

run-20251014-143022

DEV-00847 · fw v2.3.1 · 14:22

VIEW →

failed

run-20251014-140311

DEV-00391 · fw v2.3.1 · 14:03 · 2 rules fired

VIEW →

analyzing

run-20251014-133812

DEV-00512 · fw v2.2.9 · 13:38

VIEW →

What it does

Six concrete jobs, one backend.

No telemetry constellations. No insight foundries. The actual surface area, in the order you'll meet it.

☉

Ingest

Accept run manifests and NDJSON artifacts from Astrolabe via presigned URLs. Idempotent by run_id; resumes partial uploads; verifies SHA-256 checksums end-to-end.

POST /api/v1/runs/upload

♀

Validate

Rules in YAML, versioned with your test plan. Threshold, regression, schema, and shape checks. Three severities. Pinned to the plan that ran them so old runs stay reproducible.

rules/temp_drift.yaml · severity: critical

♃

Analyze

Claude Sonnet inspects every run against a golden baseline. Surfaces likely root cause, which rule fired, and what changed since the last good run. Output is grounded in the actual artifacts.

analysis.json · cites artifact line:column

☿

Query

Filter by project, device, firmware, status, rule fired, plan version, or any manifest field. Save the view; share the URL. Export the result set as CSV or JSONL without leaving the page.

?project=prod&fw=v2.3.*&rule=temp_drift

⊕

Devices

Every serial number gets a record. Firmware history, calibration events, every run it has ever produced. Open one device and see its lifetime pass rate alongside its current state.

DEV-00847 · 412 runs · last: 9m ago

♄

Report

Sign and freeze a release record. PDF and JSON; immutable; permalinked back to the exact manifests, artifacts, and rules that produced it. Re-run the analysis at any time — the record will not move.

release-2.3.1.pdf · sha256 verified

Inside the dashboard

A real surface, not a constellation.

What you see when you open a project: the runs that landed today, the rules that fired, the analysis Claude produced for each failure. Three panes, one source of truth.

Dashboard Runs · 1,247 Reports Test plans Rules Devices · 312

project · prod-validation

Recent runs

last 24h →

Run	Device	FW	Plan	Status
run-20251014-143022	DEV-00847	v2.3.1	plan-thermal-v4	done
run-20251014-140311	DEV-00391	v2.3.1	plan-thermal-v4	failed
run-20251014-133812	DEV-00512	v2.2.9	plan-power-v2	analyzing
run-20251013-174501	DEV-00847	v2.2.9	plan-thermal-v4	done
run-20251013-163200	DEV-00110	v2.2.8	plan-power-v2	done
run-20251013-141822	DEV-00391	v2.2.8	plan-thermal-v4	done

Rules fired today

3

CRITICAL

temp_drift_exceeds_2σ

DEV-00391 · run-…140311 · channel 3 · Δ +4.2°C

WARNING

stall_recovery_slow

DEV-00391 · run-…140311 · 312ms vs 180ms baseline

INFO

new_firmware_observed

DEV-00847 · fw v2.3.1 first seen 14:22

Analysis · run-…140311

claude sonnet

Channel 3 thermal sensor crossed +4.2°C above the plan-thermal-v4 golden baseline at t=812s, well outside the 2σ band. Stall recovery on the same device ran 1.7× slower than the last good run of the same firmware. The two events are likely correlated: thermal protection appears to be throttling the recovery loop.

              evidence: artifact://channels.ndjson L8,144–8,210 · L9,002
            

Data model

What Orrery stores, plainly.

Seven entities. Every screen, every API call, every report is a view over them.

☉Run

id str project str device_id str firmware semver plan_id str status enum started_at ts finished_at ts manifest_sha256 hex

☽Manifest

run_id fk env json inputs json capture_source enum astrolabe_version semver schema_version int

⊕Artifact

run_id fk kind enum s3_uri str sha256 hex bytes int rows int?

♂Device

id str serial str model str firmware_history [semver] first_seen ts runs_total int

♃Plan

id str version semver rules [fk] golden_run fk? channels [str]

♀Rule

id str kind enum severity enum spec yaml plan_version semver

♄Report

id str scope enum runs [fk] signed_by str signed_at ts pdf_sha256 hex

Rules & analysis

A rule fires. An analysis lands. The audit trail writes itself.

Rules are declarative YAML, versioned with your test plan. When one fires, Claude reads the actual artifact rows and explains what changed — in plain language, citing the data.

The rule

rules/thermal/temp_drift.yaml

# Critical: channel temperature must not drift
# more than 2σ above the plan's golden run.
id:        temp_drift_exceeds_2σ
kind:      regression
severity:  critical
plan:      plan-thermal-v4

match:
  channel:    "channels.ndjson:temp_c"
  where:      "phase == 'soak'"

assert:
  type:       band
  baseline:   plan.golden_run
  tolerance:  2     # σ
  duration_ms: 500

explain: |
  Sensor drift on this channel during soak phase
  usually indicates thermal protection or sensor aging.
  Cross-check stall_recovery_slow on the same run.

The analysis

runs/run-…140311/analysis.json

{
  "run_id": "run-20251014-140311",
  "verdict": "failed",
  "rules_fired": [
    "temp_drift_exceeds_2σ",
    "stall_recovery_slow"
  ],
  "summary": "Channel 3 crossed +4.2°C above
    golden baseline at t=812s, outside the 2σ band.
    Stall recovery ran 1.7× slower on the same run.",
  "likely_cause": "thermal throttling cascading
    into the recovery loop",
  "evidence": [
    {
      "artifact": "channels.ndjson",
      "lines": "8144-8210",
      "value": 74.2
    },
    {
      "artifact": "channels.ndjson",
      "line": 9002,
      "metric": "stall_recovery_ms",
      "value": 312
    }
  ],
  "compared_to": "run-20251013-174501",
  "signed_at": "2025-10-14T14:04:11Z"
}

Integration

Three endpoints carry the whole product.

Astrolabe talks to these so you don't have to. They're documented because everything else in Orrery sits on top of them — the dashboard, the rules engine, the reports.

POST

/api/v1/runs/upload

Astrolabe requests a presigned URL, then streams the manifest and NDJSON artifacts directly to S3. Idempotent on run_id.

→ POST /api/v1/runs/upload
{
  "run_id": "run-20251014-143022",
  "project": "prod-validation",
  "plan": "plan-thermal-v4",
  "manifest_sha256": "a7b3…"
}

← 200
{
  "upload_urls": { "manifest": "…",
                    "channels.ndjson": "…" }
}

GET

/api/v1/runs

Query by any combination of manifest fields. Pagination via cursor. Same filter grammar as the URL on the dashboard.

→ GET /api/v1/runs?
    project=prod-validation&
    fw=v2.3.*&
    rule=temp_drift_exceeds_2σ&
    status=failed

← 200
{
  "results": [ { "id": "run-…140311", … } ],
  "cursor": "eyJyIjoxMjQ3fQ=="
}

WEBHOOK

run.analyzed

Fires after analysis lands — downstream CI, Slack, or release scripts can branch on the verdict without polling.

{
  "event": "run.analyzed",
  "run_id": "run-20251014-140311",
  "verdict": "failed",
  "rules_fired": [
    "temp_drift_exceeds_2σ",
    "stall_recovery_slow"
  ],
  "permalink": "https://orrery…/r/…140311",
  "signed_at": "2025-10-14T14:04:11Z"
}

What changes

Before and after a shared backend.

Before

Shared drives and tribal knowledge.

Each engineer ships their own folder layout; the schema lives in someone's head.
A failed run is a screenshot in Slack and a CSV nobody can find a week later.
Regressions get caught when a customer files a ticket, not when the bench produces them.
Release sign-off is a meeting; the evidence is whatever was on screen at the time.

With Orrery

One backend, one schema, one URL per run.

Every run lands with manifest, checksums, and plan version. The schema is enforced at ingest.
A failed run has a permalink, the artifacts, the rules that fired, and a written analysis.
Regressions surface the moment the rule fires; the webhook tells CI before the engineer notices.
Release sign-off is a signed report. The evidence ties back to the exact bytes that produced it.

Tour Orrery in your own project.

A 30-minute walkthrough with your test data, your devices, and one of your plans wired up. No deck.

Book a tour Start with Astrolabe →