Engineering · Part 17 · onebox — cattle and pets, one payload

One deploy layer, three platforms

platform

Reference documentation for onebox: one deploy layer that runs N Django services per host, on any of three platforms — an AWS box, a rented VPS, or a Raspberry Pi behind a tunnel — from one shared payload and a small per-platform adapter. Parts 5 and 6 described the single-service ancestor; this is the system as it runs today.

Three layers, three owners

Every host has exactly three layers, and knowing which layer owns a file answers most operational questions before they're asked:

Provisioning deliberately places no config — a freshly provisioned host has an empty /etc/onebox and refuses, loudly and reachably, to boot services until the operator's files exist. That is the same invariant as "no config baked into the AMI", kept across platforms.

The manifest is the API

One JSON file declares what the host runs:

{"host": {"name": "vps1", "platform": "vps", "caddy": true},
 "services": [
   {"name": "building", "kind": "django", "domain": "app.example.com",
    "port": 8000, "backend": "local"},
   {"name": "rsi", "domain": "rsi.example.com", "port": 8010, "backend": "10.100.0.5"}
 ]}

Two consumers read it. A generator renders the Caddyfile — every service gets a TLS site, and a non-local backend becomes a reverse-proxy over the tunnel to another host (that is the whole hub→Pi mechanism). And the services bridge instantiates the unit graph: services are data, not unit files. There are five template units installed once — onebox-fetch@, onebox-venv@, onebox-release@, onebox-gunicorn@, onebox-db-init@ — and starting onebox-gunicorn@building pulls that service's whole chain by Requires=. Adding a service to a host is a manifest edit, not a provisioning change.

One boot graph

graph TD
    identity["onebox-identity
swapfile (EVERY boot) + platform identity"]:::p
    secrets["onebox-secrets
aws: SSM → env files
vps/pi: VALIDATE operator files"]:::p
    db["onebox-db
archiving drop-in (EVERY boot);
restore-or-init; arm needs-seed
if the repo has 0 backups"]:::p
    dbinit["onebox-db-init@svc
role + database"]:::s
    fetch["onebox-fetch@svc
aws: s3 cp · vps/pi: consume the
streamed artifact (or reuse current)"]:::s
    venv["onebox-venv@svc
OFFLINE uv install
(PrivateNetwork=yes)"]:::s
    release["onebox-release@svc
migrate + collectstatic + flip symlink"]:::s
    gunicorn["onebox-gunicorn@svc"]:::s
    seed["onebox-db-seed
first backup, gated on needs-seed"]:::p
    caddy["caddy ← generated Caddyfile"]:::p
    target(("onebox.target")):::t
    identity --> db
    secrets --> db
    secrets --> fetch
    db --> dbinit --> release
    fetch --> venv --> release --> gunicorn --> target
    release --> seed
    caddy --> target
    classDef p fill:#eef,stroke:#88a
    classDef s fill:#efe,stroke:#8a8
    classDef t fill:#fee,stroke:#a88

The DB branch and every service's code branch run in parallel and join at release. Two idempotence rules in that graph were earned, not designed, and they generalise: anything an external actor can pre-create, ensure every boot (the Postgres archiving drop-in is rendered before every start, because on a VPS apt's postinst creates the cluster before we do), and gate one-shot work on observable state, not on a marker from the boot that intended to do it (the first backup is armed whenever a live database faces a zero-backup repo — needs-seed — not by a fresh-init flag that expires with a failed first boot).

Code delivery: one artifact, two transports

A release is one tarball per service: git archive of the tree plus a hash-verified wheelhouse built from uv.lock, stamped with its CPU architecture. The box builds the venv offline — uv export --frozen → uv pip install --offline --no-index --find-links wheelhouse, inside a PrivateNetwork=yes unit that physically cannot reach a package index. A bad artifact fails before the live symlink moves; the previous release keeps serving.

Delivery is the platform seam. On AWS, the box pulls releases/<svc>.tgz from a bucket, and CI triggers the deploy through one fixed SSM document (it can trigger the deploy script — nothing else). Off AWS there is no bucket and no cloud credential anywhere: CI (or a laptop) streams the artifact over SSH to a deploy account whose key is locked to a forced command —

ssh deploy@host onebox-deploy building < building.tgz

— which validates the service name against the manifest, lands the tarball, and runs the same deploy script through one narrow sudo rule. Both transports end in the same place: fetch → offline venv → migrate → health-check, rolling back the symlink on failure.

The platform adapter is four verbs

Everything platform-specific fits in four small dispatch points, each a case on one variable:

Backups follow the same seam without extra code: pgBackRest's repo is S3 on AWS and a plain directory on a pet (mount a disk there); restore-or-init at boot is identical, so a wiped host rebuilds itself from its repo on every platform.

One payload, proven everywhere

The part that keeps three platforms honest: the installer that lays down the platform layer is one script, run verbatim by the AMI bake, by the VPS/Pi provisioner, and by the Docker boot harness — so none of them can drift. The harness boots the real systemd graph (PID-1 systemd, real Postgres, real pgBackRest) in two shapes: the AWS shape (fresh image, faked control plane that also records every AWS call) and the Debian shape (cluster pre-created by apt, artifacts streamed, a fake aws that asserts it was never called). Scenarios cover fresh boot, restore, plain reboot (the release-reuse path), one-service failure isolation, and graceful terminate. When a real host still finds something the harness didn't — and real hosts do — the fix lands together with a harness scenario shaped like the reality that caught it.

← All posts