Engineering · Part 7 · three independent layers

Cost and abuse control

platformai

Reference documentation for defending a service against a bill it never agreed to: a per-user denial-of-wallet budget in the app, a token kill-switch in the cloud, and an egress cap on the box — three independent layers, because any one can be bypassed.

The moment your app calls a metered, paid dependency on a user's behalf — an LLM, a media CDN, anything billed per unit — you have a new failure mode that has nothing to do with correctness: a loop, an abuser, or a leaked key can run up an unbounded charge while every request returns 200 OK. Guard it in depth, at three layers that fail independently.

flowchart TD
    subgraph InApp ["Layer 1 · in the app (per user, real-time)"]
      B["budget.check() before every paid call"]
    end
    subgraph Cloud ["Layer 2 · in the cloud (account-wide, out-of-band)"]
      K["metric alarm on usage → detach the permission"]
    end
    subgraph Box ["Layer 3 · on the box (bandwidth)"]
      E["egress rate cap via traffic control"]
    end
    B -. "caps normal use, per user" .-> Cloud
    K -. "catches a bypassed app / leaked key" .-> Box
    E -. "bounds bulk data exfiltration" .-> Done((bounded bill))

Layer 1 — a per-user budget, checked before every call

The first and finest-grained control lives in the application, on the hot path: before every paid call, ask "is this user (and the account as a whole) within budget?" and refuse if not. Track usage in your own records — you are persisting each call's token/unit counts anyway — and check several windows at once:

def check(user):
    now = timezone.now()
    # short windows are TOKEN counts — they throttle abuse and runaway loops
    if per_min  and tokens(usage(user, now - minutes(1))) >= per_min:  raise Budget('minute', …)
    if per_hour and tokens(usage(user, now - hours(1)))   >= per_hour: raise Budget('hour', …)
    # the long window is a DOLLAR cap — output dominates spend, estimate from per-unit prices
    if monthly  and cost_usd(usage(user, now - days(30))) >= monthly:  raise Budget('month', …)
    # an ACCOUNT-WIDE dollar ceiling across ALL users — the backstop before a key can drain you
    if gcap     and cost_usd(usage_all(now - days(30)))   >= gcap:     raise Budget('global', …)

Four windows, three intents. Per-minute and per-hour token caps stop a single user hammering the service or a stuck loop burning through calls — they trip fast and reset fast. The monthly dollar cap per user is the real spend limit, computed by estimating cost from per-unit prices (for an LLM, output tokens dominate, so weight them). The global monthly dollar ceiling across all users is the one that trips before a compromised credential or a viral-abuse event can empty the account — it blocks everyone, which is exactly what you want in that emergency. Keep the raw counts and prices inside this one module; what escapes is a friendly per-bucket message the caller can show.

Pair the blocking check with a non-blocking meter for the UI: the same buckets expressed as fill fractions, surfaced only once a user is near a limit. The product should feel limitless right up to the point where it must not — a usage bar that appears at 60% and reddens toward 100% communicates the ceiling honestly without making every user feel rationed.

Layer 2 — a cloud kill-switch the app cannot bypass

Layer 1 has a blind spot by construction: it only runs when the request goes through your app. A leaked cloud credential, a compromised box, or a bug that calls the paid API outside the budgeted path all sail straight past it. So add a second control out of band, in the cloud provider itself, watching the provider's own usage metric.

The pattern: a metric alarm on the dependency's usage counter (for an LLM, the output-token count over a window) whose action, on breach, is to remove the box's permission to call the dependency at all — an alarm → notification → a tiny function that detaches the invoke policy from the instance role. The paid API starts returning access-denied; the bill stops.

# CloudFormation, in words:
# Alarm:  SUM(output_token_count for this model) over N minutes > threshold
#           → SNS topic → KillLambda
# KillLambda: iam detach-role-policy  --role <InstanceRole> --policy-arn <InvokePolicy>
# Reset (manual, deliberate): re-attach the policy once you've understood the spike.

Two details make it work in practice. First, the trip is account-wide and deliberately blunt — it stops all usage, and re-enabling is a manual human act, because a kill-switch that auto-resets is not a kill-switch. Second, an alarm needs a concrete metric dimension (the specific model id), so wire it to exactly the dependency you use, and make it optional: if the feature is off (a different backend, no paid dependency), the alarm simply is not created. This layer is cheap, entirely outside your code, and it is the one that saves you when Layer 1 has been bypassed rather than merely exceeded.

Layer 3 — cap egress on the box

Some cost and some risk is not per-call at all: bulk data leaving the box — a scraped media bucket, an exfiltration attempt — is billed per gigabyte and bounded only by bandwidth. Put a ceiling on it at the operating-system level with traffic control, so the box physically cannot transmit faster than a configured rate:

# a boot-time unit shapes the outbound interface with a token-bucket qdisc
tc qdisc add dev "$iface" root tbf rate "$EGRESS_MAX_RATE" burst 32kbit latency 400ms
# rate comes from config (e.g. 10mbit); "off" removes the cap. Fail-open:
# if shaping errors, the box serves uncapped rather than not at all.

This bounds the worst case — the maximum data that can leave in an hour is now arithmetic, not "however fast the NIC goes" — and it composes with a matching cloud alarm on the media bucket's download bytes, so an egress spike both throttles (on the box) and pages you (in the cloud). Note the deliberate fail-open stance here, the opposite of your config's fail-closed: a bug in the bandwidth shaper should degrade to "serves at full speed", not "serves nothing", because availability outranks the cost cap for this particular control. Knowing which way each safety mechanism should fail is half of designing them.

Why three, not one

Each layer covers a different threat and has a different blind spot, which is the entire justification for the redundancy:

Delete any one and a specific, plausible incident becomes unbounded. Together they turn "a bad day could cost an unbounded amount" into "a bad day costs at most a known, small amount", which is the only acceptable answer when the downside is your own bank balance.

The shape to copy

1. In-app: a check() before every paid call — fast token windows for abuse, a monthly dollar cap per user, a global dollar ceiling across all users; a friendly UI meter from the same buckets.
2. In the cloud: a usage-metric alarm that detaches the invoke permission; blunt, account-wide, manual reset; optional so it only exists when the paid feature does.
3. On the box: a fail-open egress rate cap, paired with a cloud egress alarm.
4. Decide, per control, which way it fails — the budget and kill-switch fail closed, the bandwidth shaper fails open.

Part 8 is the configuration discipline underneath all of this — fail-closed settings with no defaults, and how "explicitly off" survives a store that cannot hold an empty string.

← All posts