Engineering · Part 10 · the media pipeline

Eager, pluggable rendering

platform

Reference documentation for turning uploaded files into viewable derivatives — page images and a thumbnail contact-sheet — eagerly on upload, through a single pluggable backend that renders in-process for tests and on a serverless function in production, with one set of pure transforms that cannot drift between the two.

Users upload PDFs and images; the UI needs fast thumbnails for a grid and a crisp full-size view per page. The naive approach — resize on every request — is slow, uncacheable, and repeats the expensive work forever. This document describes the opposite: do the expensive work once, when the file arrives, store the results, and serve static derivatives thereafter. The design tension is that "once, on upload" means different things in a test, on a laptop, and in production — solved by making the expensive step a swappable backend behind a stable interface.

Eager, once, into two shapes

On upload, render each page to two things and store them:

a full-size base — the longest side capped (say 2048px) — for the per-page view and, later, as model input for any AI feature;
a sprite — every page's small tile (say ≤200px) packed into one contact-sheet image — so a grid of N thumbnails is one HTTP request and one decode, not N.

The sprite is the detail worth stealing: a document grid that would be dozens of thumbnail requests becomes a single image with CSS background-position offsets per tile. It renders instantly and caches as one object. Everything downstream serves these static derivatives; the original file is never resized on a request again.

The pure core: transforms that cannot drift

Separate the pure image work from everything else, in a module that depends only on an imaging library — no framework, no database, no cloud SDK. It takes and returns bytes and image objects, nothing more:

# imaging_core.py — pure; the SAME code runs in-process and in the serverless function
BASE_PX, TILE_PX = 2048, 200

def page_count(data):            -> int
def render_page(data, page)      -> bytes        # one page → a capped base image
def make_tile(base_img)          -> image        # a base → a small tile
def pack_sprite(tiles)           -> (bytes, cols, tile_w, tile_h)   # tiles → one contact-sheet

Because this module is pure and dependency-light, the identical code runs in your test process, in a local subprocess, and inside a serverless function — so a page can never render differently in production than it did in the test that approved it. It is also trivially unit-testable: bytes in, bytes out, no fixtures. Keep it ruthlessly free of environment concerns; that purity is what lets the backend below be swappable at all.

One pluggable backend, three environments

"Render on upload" has three correct implementations depending on where you are. Select between them with one config value, so the call site — the upload handler — is identical everywhere:

flowchart TB
    Up["upload handler calls render.start(asset)"]:::u --> Sw{"RENDER_BACKEND"}:::s
    Sw -- inline --> I["render synchronously,
same process + txn (tests)"]:::a
    Sw -- subprocess --> P["Popen a detached worker
(local dev; upload returns now)"]:::b
    Sw -- lambda --> L["enqueue a message →
serverless renders → signed callback"]:::c
    classDef u fill:#eff6ff,stroke:#1d4ed8;
    classDef s fill:#fffbeb,stroke:#b45309;
    classDef a fill:#ecfdf5,stroke:#047857;
    classDef b fill:#f3f4f6,stroke:#374151;
    classDef c fill:#fef2f2,stroke:#b91c1c;

inline — render synchronously, in the same process and database transaction. This is for tests: deterministic, no async, the derivatives exist the instant the upload call returns, so a scenario can assert on them immediately.
subprocess — spawn a detached worker process and return the upload response immediately; the derivatives appear a moment later. This is for local development: real asynchrony, no cloud, the UI shows "rendering…" placeholders that poll.
lambda — enqueue a message and return; a queue-triggered serverless function renders into object storage and calls back. This is production: the expensive, memory-hungry work is off the web box entirely, scales to zero, and a burst of uploads does not starve request-serving.

The upload handler never knows which backend is active — it calls one start(asset). The asset carries a status (pending → processing → done) that the UI polls, so all three backends present the same "rendering, then ready" experience; only the latency and the machinery differ.

The production path: queue, function, signed callback

The serverless backend is worth detailing because its wire contract is the part you must get right. On upload you enqueue a small message and the function does the rest:

# enqueue: {asset_id, original_key, page_count, callback_url}
# the callback_url already carries a signed token (below)

# the function (running imaging_core): read the original from storage → render each
# page's base + the sprite → write them to storage → POST the callback per page and at finish.

The callback is how the function tells your app "page 3 is ready" / "all done", and it crosses the public internet, so it must be authenticated without a shared session. Sign it with an HMAC the app can verify — a token derived from the asset id and a secret key, compared in constant time:

def callback_token(asset_id):
    return hmac.new(RENDER_CALLBACK_KEY.encode(), str(asset_id).encode(), 'sha256').hexdigest()

def verify_callback(asset_id, token):
    return hmac.compare_digest(callback_token(asset_id), token or '')

# the callback endpoint:  /render/callback/<asset_id>?s=<token>  → verify, then mark ready

The token travels in the callback URL you put on the queue message, so the function needs no credentials of its own — it just POSTs back the URL it was given. A wrong or missing token is rejected; an attacker cannot forge "rendering complete" for an arbitrary asset. Use constant-time comparison so the check does not leak the token byte by byte.

Failure and idempotency

Because the queue invokes the function directly, a bad render fails into the queue's machinery, not a user's face: the message retries, and after a few failures lands in a dead-letter queue with an alarm — you find out, the user sees a stuck "processing" they can retry, and nothing is silently lost. Make the render idempotent (writing the same derivatives for the same asset is harmless) so a retry is always safe, and guard the state transition (pending → processing) atomically so two triggers cannot double-render. The same discipline lets a boot-time sweep re-enqueue anything left pending after a crash.

Serving the derivatives

Serve the finished derivatives access-controlled, exactly as you would any tenant-scoped resource: the base and sprite belong to the uploading tenant, so a request for them goes through the same membership check as the rest of that tenant's data (Part 1). Locally, stream them from disk; in production, redirect to a short-lived signed storage URL so the bytes come straight from object storage without proxying through the web box. The render pipeline produced static objects; access control decides who may fetch them.

The shape to copy

Render eagerly on upload into two shapes: a capped full-size base per page, and one packed sprite for grids.
Put the pure transforms in a dependency-light module (bytes in, bytes out) so the identical code runs in tests and in production.
One start(asset) call site; a config value selects inline (tests) / subprocess (dev) / serverless (prod); the asset's status makes all three feel the same to the UI.
In production: enqueue → queue-triggered function → HMAC-signed callback; retries → dead-letter queue + alarm; idempotent renders and an atomic state transition.
Serve derivatives access-controlled — stream locally, signed-URL redirect in production.

Part 11 returns to the front end for one deliberately small thing done consistently: a single account widget that is identical across the marketing site and the signed-in app.

← All posts