Rate limiting when you're the client, not the server

Most writing on API rate limiting teaches you to defend your own endpoints. An agent system spends most of its life on the other side: making calls into Gmail, Slack, Stripe and a thousand others, each with different limits, different headers, and different opinions about what you did wrong. Being a well-behaved client is a distinct engineering problem.

Why agents hit limits harder

A user clicks a button and makes one request. An agent given “clean up my inbox” makes hundreds, as fast as your code allows, in a burst that looks exactly like abuse.

Three things make it worse than normal API consumption. Bursts are unpredictable, because they depend on what the agent decides to do. Retries multiply, because a naive retry on a 429 is a request that arrives when you’re already over. And limits are shared across your users when you hold an app-level quota, so one heavy run degrades everyone.

Read the response, don’t guess

Most providers tell you what’s happening. Use it before you build anything clever.

Retry-After is authoritative — honour it exactly. If it says 30 seconds, waiting 5 with exponential backoff is both wrong and rude. Note it comes in two flavours: delay-seconds and an HTTP-date.

X-RateLimit-Remaining and X-RateLimit-Reset let you throttle before you get rejected, which is far better than reacting after. When remaining runs low, slow down rather than sprinting into the wall.

Also distinguish the two 429s people conflate: rate limiting (too fast, retry shortly) and quota exhaustion (out of budget for the period, retrying today won’t help). Backing off repeatedly against an exhausted daily quota just burns time and looks like an outage.

Token bucket, per credential

For your own throttling, a token bucket is the right primitive: it permits controlled bursts while bounding the sustained rate, which matches how real limits work.

The key detail is what you bucket by. Per provider is too coarse — one user’s heavy run starves everyone. Per credential is usually right, since that’s the unit the provider limits. And a global ceiling per provider on top, if you hold an app-level quota.

Keep the bucket in shared state (Redis, typically) if you run more than one instance. Per-process buckets multiply your effective rate by your instance count, which is how teams discover they’re rate limited only after scaling up.

Jitter is not optional

Pure exponential backoff synchronises clients. Fifty requests fail at once, all wait exactly 1s, all retry at once, all fail again. That’s a thundering herd of your own making.

Full jitter — sleep = random(0, base * 2^attempt) — spreads them out and consistently outperforms both fixed and “equal jitter” variants in practice. It’s one line, and it’s the difference between recovering and oscillating.

Cap the exponent, too. Unbounded doubling reaches absurd delays, and by then you should be failing the run rather than sleeping for an hour.

Retry the right failures

A retry policy that retries everything is an amplifier, and it’s the single most common way a small failure becomes an incident.

Retry: 429, 502, 503, 504, connection resets, timeouts. Transient by nature.

Don’t retry: 400, 401, 403, 404, 422. A malformed request stays malformed. Retrying a 401 three times just triples your auth failures — and some providers count those toward lockouts.

Also honour idempotency. Retrying a POST that already succeeded but whose response you lost can double-charge someone. Use the provider’s idempotency key mechanism where it exists; where it doesn’t, prefer at-most-once semantics for anything that spends money or sends messages.

Queue instead of blocking

When an agent is rate limited mid-run, sleeping in the request handler is the wrong shape — you’re holding a connection and a worker to do nothing.

Better: push the pending call onto a queue with a scheduled retry time, release the worker, and resume the run when it fires. This means agent runs need to be resumable — checkpointable state rather than a stack frame — which is a design decision worth making early because retrofitting it is painful.

It also makes the honest UI possible: “waiting on Gmail’s rate limit, retrying in 40s” rather than a spinner that looks broken.

Make it visible

Rate limiting is invisible until it isn’t. Instrument it:

• 429s per provider, per credential
• Time spent waiting, per run
• Queue depth and oldest waiting item
• Retry counts, so an amplifier shows up as a spike

The metric that matters most is wait time as a share of run duration. When that climbs, your agents aren’t slow — they’re queuing, and the fix is throttling earlier rather than optimising code.

The checklist

1. Honour Retry-After exactly, both formats.
2. Throttle proactively on remaining/reset headers.
3. Token bucket per credential, plus a global provider ceiling.
4. Shared bucket state across instances.
5. Full jitter on backoff, with a capped exponent.
6. Retry only transient classes; never 4xx-validation.
7. Idempotency keys on anything that spends or sends.
8. Queue and resume instead of blocking a worker.
9. Distinguish rate limits from quota exhaustion.

Items 4 and 5 are the ones that surprise people, because both work fine until you scale out.

See also: cutting the calls you didn’t need to make and handling duplicate webhook deliveries.