Webhook idempotency: handling the same event twice

Webhook idempotency is the property that receiving the same event twice has the same effect as receiving it once. You need it because every serious provider delivers at least once — and a retry after a slow-but-successful handler is normal traffic, not an error. Here’s the shape that holds up, and the two mistakes that make it fragile.

Why duplicates are guaranteed

A provider sends your endpoint an event and waits for a 2xx. If your handler takes eight seconds and their timeout is five, they mark it failed and retry — even though you completed it. Nothing is broken; you simply have no way to tell them so retroactively.

Network partitions do the same. So does a deploy mid-request. Assume duplicates and the problem becomes routine; assume exactly-once and you get double-charged customers.

Verify the signature first

Before anything else. A public webhook endpoint is an unauthenticated command channel — whoever can POST to it can tell your system that things happened.

The concrete risk depends on the event. For a bounce webhook, a forged event lets an attacker permanently suppress any address they choose, which is a targeted denial of service against individual users. For a payment webhook, it’s worse.

Most providers use an HMAC over id.timestamp.body:

const expected = createHmac("sha256", secret)
  .update(`${id}.${timestamp}.${rawBody}`)
  .digest("base64");

Two details people get wrong:

Use the raw body. Parsing JSON and re-serialising changes bytes — key order, whitespace, unicode escapes — and the signature won’t match. Read the raw text, verify, then parse.

Check the timestamp. Without a tolerance window, a captured valid request replays forever. Reject anything older than a few minutes.

And compare in constant time. A fast-exit comparison leaks the expected signature byte by byte.

Key on the provider’s event id

The core of idempotency is a table of events you’ve already processed, keyed by the provider’s id — not by your own generated one, which would be different each delivery.

insert into email_events (id, type, payload)
values (p_event_id, p_type, p_payload)
on conflict (id) do nothing;

if not found then
  return 'duplicate';   -- already processed; do nothing else
end if;

The primary key does the work. The second delivery conflicts, inserts nothing, and returns early. No locking scheme, no distributed coordination.

Do the insert and the effect in one transaction

This is the mistake that survives review, because the naive version looks correct.

If you record the event, then apply its effect as a separate statement, there’s a window between them. A crash there leaves the event marked processed and the effect never applied — and because it’s marked processed, the retry does nothing. The event is lost permanently, which is worse than processing it twice.

Put both in one transaction:

insert into email_events ... on conflict do nothing;
if not found then return 'duplicate'; end if;

update email_log set status = v_status where resend_id = p_resend_id;

if p_type in ('email.bounced','email.complained') then
  insert into email_suppressions (email, reason) values (p_email, ...)
  on conflict (email) do update set reason = excluded.reason;
end if;

Either the event is recorded and applied, or neither happened and the retry gets a clean attempt.

Return the right status code

Your status code is instruction to the sender.

• 2xx — done, stop retrying. Return this for duplicates too; they are handled.
• 4xx — malformed or unverifiable. Don’t retry; a bad signature won’t improve.
• 5xx — transient failure. Please retry.

Getting these backwards causes real damage. Returning 200 on a database failure silently drops the event, and the provider will never mention it again. If you couldn’t apply it, say 500 and let them retry — that retry is your recovery mechanism.

Out-of-order delivery

Retries mean events can arrive out of order: a delivered retry landing after a later bounced.

If your state machine only moves forward, guard the transition rather than blindly assigning. If ordering genuinely matters, store the provider’s timestamp and ignore events older than the state you already hold.

The checklist

1. Verify the signature over the raw body, in constant time.
2. Reject stale timestamps.
3. Insert on the provider’s event id with on conflict do nothing.
4. Return early on duplicates — with a 2xx.
5. Record and apply in one transaction.
6. Use 4xx for unverifiable, 5xx for transient.
7. Guard state transitions against out-of-order arrival.

Steps 1 and 5 are the ones that cause incidents. The rest are hygiene.

See also: how we store one-time codes — same atomicity principle, different problem.