Engineering · Part 6 · fast recovery over high availability

Infrastructure as code, and deploy on merge

platform

Written before the onebox split. The deploy layer described below was since generalised into onebox (ONEBOX.md): the same box now runs N services and provisions identically onto AWS, a bare VPS, or a Pi — no bucket or cloud credentials on the zero-AWS paths. The principles here (fast recovery over high availability, restore-on-boot, one reconciling admin command) all survived; the specifics — the infra/ tree, app.tgz, the per-app unit names — are this document's era.

Reference documentation for shipping a single-box service to AWS: infrastructure-as-code with CloudFormation, one admin command that reconciles everything, and a CI pipeline that deploys on merge and on tag — built on an explicit bet that fast recovery beats high availability.

Part 5 described how one machine boots and restores itself. This document is the other half: how the machine, its surrounding cloud resources, and the code that runs on it all get created and updated — reproducibly, from a laptop or from CI, with the same commands.

The bet: recover fast, don't replicate

Before any YAML, decide your availability model, because it dictates the whole shape. This design makes a deliberate, unfashionable choice: one box, no load balancer, no read replica. Postgres runs on the same instance as the app; there is a single elastic IP; there are no redundant copies of anything hot.

You pay for that with a non-zero recovery time — if the box dies, a replacement takes a few minutes to launch and restore the database from backup. You are not paying for a near-zero recovery point (no data loss) via synchronous replicas, nor for the operational weight of a multi-node cluster. Three things make the trade safe:

• Continuous backups + restore-on-boot (Part 5): the database branch restores from object storage before the app is allowed to start, so recovery is the boot path, exercised on every replacement — not a runbook nobody has tested.
• An auto-scaling group of size one. The ASG's only job is to keep exactly one healthy instance alive: when the box self-reports unhealthy (Part 5), the group terminates it and launches a replacement, which restores itself. You get self-healing without a load balancer.
• Hardware cheap enough to be disposable. A tiny Graviton instance costs pennies, so "throw the box away and rebuild it" is a normal operation, not a budget event.

Choose this when a few minutes of downtime a few times a year is survivable and simplicity is worth more than nines — a great many products qualify. Choose replication instead when downtime has per-second cost. The rest of this document assumes the former.

Infrastructure as code — five stacks, split by lifecycle

Everything AWS-side is CloudFormation. The important decision is not "use CloudFormation" — it is where you draw the stack boundaries, and the rule is: split by how long a thing lives and who may destroy it.

Five templates, deployed in dependency order — the data stack first (nothing runs without its buckets), then compute, then the image pipeline, with the two GitHub stacks standing apart in the CI account:

Four buckets, separated by who owns and who may destroy them

The data stack is not one bucket but four, split by contents so their access rules and lifecycles never entangle. All four are Retain-on-delete and deny non-TLS access; what differs is who writes them and what a leak would expose:

Keeping deploy artifacts (code) apart from user uploads (media) is the important separation: they have different writers (your release pipeline vs your application), different blast radii (a leaked release is public code; leaked media is user data), and different retention needs. Bundling them would force one access policy onto two very different risk profiles. The app stack receives all four names as parameters (from the data stack's outputs) and never creates a bucket itself — so the compute stack can be destroyed and recreated with the buckets, and everything in them, untouched.

The payoff of the data/app/image split: you can delete and recreate the entire compute stack — to change instance type, fix a broken launch template, or start clean — without endangering a single byte of user data or a single backup, because those live in a Retain-policy stack the app stack only references by output. The recreated box restores its database from the untouched backup bucket on boot. The "recover fast" bet and the stack boundaries are the same idea expressed twice.

The admin path: platform.sh

One command, run by an administrator with infrastructure credentials, brings an entire environment (staging or production) to its declared state. It is idempotent — run it as often as you like — and it is the only path that touches infrastructure or bakes images. Five steps:

flowchart TD
    S1["1 · reconcile config
generate + write secrets/params to the param store"]:::a
    S1 --> S2["2 · deploy CloudFormation
data → app → image stacks, in order"]:::b
    S2 --> S3["3 · stack outputs → param store
bucket names, domain, queue URLs"]:::a
    S3 --> S4{"4 · AMI current?
compare baked vs declared version"}:::c
    S4 -- "changed / none" --> Bake["bake a new AMI
+ start an instance-refresh"]:::c
    S4 -- "unchanged" --> S5
    Bake --> S5["5 · validate + smoke
DNS, TLS, Stripe/SES keys, the real user path"]:::d
    classDef a fill:#f3f4f6,stroke:#374151;
    classDef b fill:#eff6ff,stroke:#1d4ed8;
    classDef c fill:#fffbeb,stroke:#b45309;
    classDef d fill:#ecfdf5,stroke:#047857;

Two design points inside those steps are worth copying:

• Config is reconciled before compute, and flows one way. Step 1 writes every setting to a parameter store; step 2 launches boxes that read it at boot (Part 5); step 3 writes the stack's own outputs (the bucket names it just created, the queue URL) back to the param store, because the next boot needs them. Ordering matters: a box launched in step 2 that boots before step 3 must still find its bucket names — so the data-stack outputs are written in step 2, immediately after that stack, not batched at the end. Get this wrong and a fresh boot races a half-written config.
• The AMI rebake is version-driven, not a flag you must remember. The image template carries a version number; each baked AMI is stamped with the version it was built from. Step 4 bakes only when there is no AMI yet, or when the template's version is newer than the baked one — i.e. when what is baked (OS packages, the systemd units, the deploy script) actually changed. Application code is never in the AMI, so shipping a bug fix never triggers a bake. When a bake does happen, it rolls the new image in via an ASG instance-refresh, and the fresh box boot-fetches the current code.

Wiring CI to AWS without long-lived keys: github.sh

A second admin command connects the repository to an environment's AWS account — once — so that CI can deploy with no stored credentials anywhere. It creates an OIDC identity provider and a per-environment release role, then creates the matching GitHub Environment and sets the env-scoped values the deploy workflow reads:

gh api -X PUT repos/$REPO/environments/$ENV                       # the GitHub Environment
gh secret   set AWS_RELEASE_ROLE_ARN --env $ENV --body "$ROLE"    # the role CI assumes
gh variable set RELEASE_BUCKET       --env $ENV --body "$BUCKET"  # where artifacts land
gh variable set APP_URL              --env $ENV --body "https://$DOMAIN"  # for the smoke test
gh secret   set SMOKE_PASSWORD       --env $ENV --body "$PW"

At deploy time CI presents a short-lived OIDC token; AWS exchanges it for temporary credentials for the release role, but only for a job whose OIDC subject matches repo:…:environment:production. The role itself is deliberately tiny — it can put a deploy artifact and issue one run-command to the box, nothing more. It cannot change infrastructure, networking or images; that stays on the admin's platform.sh path. Compromising CI cannot reshape your cloud.

Two deploy halves, one command each

Shipping code is split into two independent halves so a fix to one never rolls the other. Both run identically locally and in CI — the CI job literally invokes the same script.

# the CODE half — the box
./infra/bin/deploy-app.sh <env>
#   git archive HEAD + a prebuilt arm64 wheelhouse → app.tgz
#   → upload to the code bucket → one run-command to the box:
#     code-deploy.sh builds a fresh venv beside the current one, migrates,
#     collects static, flips an atomic symlink, and rolls back on any failure.

# the RENDER half — a serverless function (see Part 10)
./infra/bin/deploy-render.sh <env>
#   build the arm64 render.zip (+ a native smoke that gates the ship)
#   → upload to a FIXED key in the code bucket
#   → point the function's $LATEST at it. No versions, no alias:
#     the queue invokes $LATEST; a bad deploy fails into the dead-letter queue.

The render function is deployed by simply repointing $LATEST; there is no versioned-alias ceremony because the queue is the only caller and a broken render lands in a dead-letter queue with retries and an alarm, not in a user's face.

Building the wheelhouse: the box never compiles

Both artifacts bundle a wheelhouse — a directory of prebuilt binary wheels for every dependency — so the target installs offline, never reaching a package index and never compiling from source at deploy time. That removes an entire class of "green in CI, fails on the box" failures (a missing compiler, a transient index outage, a source build that behaves differently). Two details make it reliable:

• Build for the target's architecture, cheaply — and never compile at all. The box is ARM (Graviton). Export the locked runtime set from the lockfile (uv export --frozen → a hashed requirements file) and run pip download --only-binary=:all: against it, which fetches prebuilt ARM wheels only — hash-verified against the lock — and fails loudly if one is missing, rather than falling back to a source build (whose locally-built wheel would not match the lock's hashes, and which is brutally slow under x86→ARM emulation and looks like a hang). On a non-ARM host, run the same download inside an ARM container.
• Reuse only a complete build. A half-finished wheelhouse (an interrupted build) would silently ship a zero-wheel artifact whose offline install fails on the box. So stamp a completed build with a hash of the lockfile, and reuse the cache only when the stamp is present and matches — never on a bare "the directory exists". CI keys its wheelhouse cache on the same lockfile hash, so an unchanged dependency set is built once and restored thereafter.

stamp=".ok-$(sha256 uv.lock | head -c16)"
[ -f "wheelhouse/$stamp" ] && return          # reuse ONLY a stamped, complete build
rm -rf wheelhouse                             # wipe any partial/stale dir first
<build natively, or download-only in an ARM container>
ls wheelhouse/*.whl >/dev/null || die 'no wheels built'   # prove ≥1 wheel landed
touch "wheelhouse/$stamp"                      # only now is it safe to reuse/ship

Deploy on merge, and on tag

The two environments have two triggers, and the difference between them is one line of workflow config plus one approval gate:

flowchart TD
    Merge["push to main"]:::m --> TS["tests
lint + unit + acceptance"]:::t
    Tag["push a v* tag"]:::m --> TP["tests
lint + unit + acceptance"]:::t
    TS --> DS(("deploy: STAGING"))
    TP --> Gate{"manual approval
(GitHub Environment)"}:::g
    Gate --> DP(("deploy: PRODUCTION"))
    DS --> AS["deploy-app.sh"] & RS["deploy-render.sh"]
    DP --> AP["deploy-app.sh"] & RP["deploy-render.sh"]
    AS --> SmS["post-deploy smoke"]
    AP --> SmP["post-deploy smoke"]
    classDef m fill:#eff6ff,stroke:#1d4ed8;
    classDef t fill:#f3f4f6,stroke:#374151;
    classDef g fill:#fffbeb,stroke:#b45309;

• Push to main → staging, automatically. The shared test workflow gates the deploy; then the two deploy halves run in parallel; then the post-deploy smoke (Part 3) drives the real user path against the just-deployed environment.
• Push a v* tag → production, behind an approval gate. Same tests, same two halves, same smoke — but the GitHub Environment requires a human to approve the run before it touches production. Tagging is the deliberate act; approval is the second pair of eyes.

The property that makes this trustworthy: the deploy is not a special CI-only path. CI runs the exact infra/bin/ scripts an engineer runs from a laptop, gated by the exact test suite that runs on every pull request. There is no "works on my machine / breaks in the pipeline" gap because there is only one mechanism, exercised from two places. The post-deploy smoke, run every time in every environment including production, is the final proof that what shipped actually serves a real request end to end.

Carrying test output into the release

One neat consequence of building artifacts in CI: a job can fold its own output into the release. The acceptance suite (Part 3) captures a screenshot gallery of every scenario; you want that gallery served by a staff page on the deployed box, as evidence of what the tests saw. So the pipeline passes it from the test job to the deploy job as a build artifact:

flowchart TB
    T["test job (tests.yml)
behave writes reports/ = screenshots + a manifest"]:::t
    T -->|upload artifact| G[("ui-test-gallery")]:::art
    T -->|cache| C[("reports-<sha>")]:::art
    G -->|download into ./reports| D["deploy-app job
build app.tgz WITH reports/ baked in"]:::d
    D --> Box["the box serves /staff/test
from the baked reports/"]:::b
    C -.->|"a v* tag on the SAME commit reuses it"| Prod["production deploy skips re-testing"]:::p
    classDef t fill:#f3f4f6,stroke:#374151;
    classDef art fill:#eff6ff,stroke:#1d4ed8;
    classDef d fill:#ecfdf5,stroke:#047857;
    classDef b fill:#fef2f2,stroke:#b91c1c;
    classDef p fill:#fffbeb,stroke:#b45309;

• The test job writes reports/ (the manifest + screenshots + copied feature files) and uploads it as an artifact named ui-test-gallery.
• The deploy-app job downloads that artifact into ./reports before building app.tgz, so the archive that ships to the box contains the gallery. The staff page then renders the real screenshots instead of a "not baked" fallback — the deployed box literally carries the evidence that its own code passed acceptance.
• The test job also caches reports/ under the commit SHA (reports-<sha>). This is the tag fast-path: a v* production tag points at a commit already merged to main and already tested, so its pipeline finds the cached gallery for that exact SHA and skips re-running the whole suite — a cache miss safely falls back to running it. The same passing run gates both the staging deploy and the later production deploy of the same commit.

The shape to copy

1. Decide the availability model first: fast recovery (restore-on-boot + ASG-of-1) vs replication. Everything else follows.
2. CloudFormation, split by lifecycle: durable data (Retain) separate from destroyable compute separate from the image pipeline.
3. One idempotent admin command reconciles config → deploys stacks → writes outputs back → version-driven AMI bake → validate + smoke.
4. Connect CI to AWS with OIDC and a narrow per-environment release role — no stored keys; the role can ship code, not reshape infrastructure.
5. Two deploy halves (code + render), one command each, identical local and in CI; bundle a target-architecture wheelhouse (built natively or download-only, stamped-when-complete) so the box installs offline; separate buckets for media / code / backups / certs.
6. Merge → staging automatically; tag → production behind approval; the same scripts, the same tests, a post-deploy smoke every time — and fold test output (the gallery) into the release artifact, cached per commit so a tag reuses its merge's passing run.

Part 7 looks at what protects this box from a bill it did not agree to: a denial-of-wallet budget, a token kill-switch, and an egress cap.

← All posts