Engineering · Part 13 · certs that survive replacement

Automatic HTTPS on a disposable box

platformsecurity

Reference documentation for automatic HTTPS on a box that gets thrown away and rebuilt: certificates that survive instance replacement by living in object storage, and a fail-safe default that makes it impossible to burn your certificate authority's rate limit while iterating.

A reverse proxy with automatic certificate management (Caddy, and the ACME protocol behind it) turns "get and renew TLS certificates" into a solved, zero-touch problem — on a permanent server. On a disposable box, the one this series has been building (Part 6: recover by replacement), the naive setup has a nasty edge: every replacement is a brand-new machine with no certificates, so it requests fresh ones from the authority on every boot. Do that a few times in an afternoon of debugging and you hit the authority's rate limit — for Let's Encrypt, five certificates per registered domain per week — and now your box cannot get a trusted certificate at all. This document is the two design choices that make automatic HTTPS and disposable boxes coexist.

The proxy config is tiny — and holds no secrets

# Caddyfile — TLS termination + reverse proxy to the app on localhost
{
    email {$ACME_EMAIL}
    acme_ca {$ACME_CA:https://acme-staging-v02.api.letsencrypt.org/directory}
    storage s3 {                       # ← the whole trick: cert state in object storage
        host "s3.{$AWS_REGION}.amazonaws.com"
        bucket "{$ACME_BUCKET}"
        use_iam_provider true          # the instance's role — no static keys
    }
}
{$APP_DOMAIN} {
    encode zstd gzip
    reverse_proxy 127.0.0.1:8000       # → gunicorn
}

Everything variable — the email, the certificate authority URL, the domain, the storage bucket — comes from the environment, so the same file works for every environment and the file itself carries no secrets. The proxy terminates TLS at the edge and forwards plain HTTP to the application on localhost, which is why the app never deals with certificates at all.

Choice one — certificate state lives in a bucket, not on the box

The fix for "a fresh box has no certificates" is to stop keeping them on the box. Point the proxy's certificate storage at an object-storage bucket — the dedicated acme bucket from Part 6 — so the ACME account key and the issued certificates persist independently of any instance:

flowchart LR
    B1["box #1 boots"]:::b --> W["proxy: certs in the bucket?"]:::q
    W -- "no (first ever)" --> Iss["request from the authority once"]:::i --> Store[("acme bucket:
account key + certs")]:::s
    B2["box #2 (a replacement) boots"]:::b --> W2["proxy: certs in the bucket?"]:::q
    W2 -- "yes" --> Reuse["reuse them — no request to the authority"]:::r
    Store -.-> W2
    classDef b fill:#eff6ff,stroke:#1d4ed8;
    classDef q fill:#fffbeb,stroke:#b45309;
    classDef i fill:#fef2f2,stroke:#b91c1c;
    classDef s fill:#f3f4f6,stroke:#374151;
    classDef r fill:#ecfdf5,stroke:#047857;

Now a replacement box finds the existing certificates in the bucket on boot and simply uses them — no request to the authority, no rate-limit consumption. The authority is contacted only for a genuine first issuance or a real renewal (well inside the limit), regardless of how many times the box itself is replaced. Authenticate the bucket access with the instance's own IAM role, so there are — consistent with the rest of this architecture — no static credentials in the proxy config. The certificate lifecycle is now decoupled from the instance lifecycle, which is exactly what a disposable-box design needs.

Choice two — a fail-safe default that cannot burn the real limit

The rate limit still bites during setup, when you are legitimately issuing for the first time and iterating. Certificate authorities anticipate this with a staging environment: same protocol, certificates that browsers do not trust, and generous limits. The design rule is to make staging the default you cannot forget:

acme_ca {$ACME_CA:https://acme-staging-v02.api.letsencrypt.org/directory}
#         ^^^^^^^^ if ACME_CA is unset/empty, fall back to STAGING — never production

Read the fallback: a missing or empty ACME_CA resolves to the staging authority, never production. This is the crucial inversion — the same default-deny instinct as Part 12. A boot-loop, a misconfiguration, a half-finished environment can request staging certificates all day and cost nothing; it is impossible to accidentally hammer the production authority and burn the five-per-week limit, because reaching production requires explicitly setting the production URL. A box is opted into real, trusted certificates as a deliberate act — "this box is good, promote it" — not as a side effect of which environment it happens to be. Both staging and production environments therefore start on untrusted certificates, and you flip an environment to real ones only when you have watched it come up healthy:

# when the box is proven good, point it at the production authority and replace the instance
param-set ACME_CA https://acme-v02.api.letsencrypt.org/directory
replace-instance      # the fresh box reads the prod authority and issues a trusted cert,
                     # reusing the same account key already in the bucket

Why the two choices need each other

Either choice alone is insufficient, which is why both are the pattern. Bucket-backed storage without the staging default means that while you are still iterating — before the certs exist to persist — a boot-loop can still burn the production limit. The staging default without bucket-backed storage means that once you are on production, every box replacement re-issues and creeps toward the limit again. Together they cover both phases: staging protects you during setup, and persisted storage protects you forever after, so automatic HTTPS is genuinely zero-touch even though the box underneath it is cattle, not a pet.

The shape to copy

Terminate TLS at an auto-certificate reverse proxy; keep its config secret-free (everything from the environment); forward plain HTTP to the app on localhost.
Store the ACME account key and certificates in an object-storage bucket, authenticated by the instance role — so a replacement box reuses them instead of re-issuing.
Default the certificate-authority URL to staging, so production issuance is only ever a deliberate, explicit opt-in — a boot-loop cannot burn the real rate limit.
Flip an environment to the production authority only after it has come up healthy, then replace the instance so it issues a trusted certificate.

Part 14 turns to the other thing that must survive a box being thrown away: its database — and how "restore from backup" becomes the boot path itself.

← All posts