Webhooks arrive twice and out of order: receivers that survive

A payment succeeds in Stripe, an order updates in Shopify, and your server is supposed to find out. The mechanism for that is a public URL you expose so the vendor can POST to it. Everything about that sentence is a problem: the URL is public, the vendor decides when and how often to call it, and the call has to be answered in seconds.

The constraint

Three things have to hold. First, delivery is retried, not guaranteed once: Stripe "attempts to deliver events to your destination for up to three days with an exponential back off," and its docs warn that endpoints "might occasionally receive the same event more than once." Second, order is not preserved: "Stripe doesn't guarantee the delivery of events in the order that they're generated," and Shopify says the same within a topic and across topics for the same resource. Third, the endpoint is on the open internet, so anyone who finds the URL can POST a payload that looks like a paid invoice.

The options

Trust and process inline. Parse the body, update the database, return 200. This is the obvious first version of the integration, and it fails on all three constraints: unverified payloads are forgeable, synchronous work blows the response deadline (Shopify allows "a one-second connection timeout and a five-second timeout for the entire request"), and a crash mid-handler loses the event while the vendor thinks it was delivered.

Poll the API instead. Skip webhooks, reconcile on a schedule. Correct but hours stale, and you trade the vendor's push problem for your own rate-limit problem. It is the same fork we walked for browser clients in websockets, SSE, or polling: simple, late, billed to you. Worth keeping as the recovery path, not the primary one: Shopify's own docs say "your app shouldn't rely on receiving data from Shopify webhooks" and recommend "reconciliation jobs to periodically fetch data." The poll is the backup, the webhook is the latency.

Queue-first receiver. Verify the signature, dedupe the event id, ack with 200, push to a queue, process async. Stripe's docs recommend exactly this shape: return "a 200 response before updating a customer's invoice as paid in your accounting system" and process "incoming events with an asynchronous queue." This is the pattern we ship.

The mechanism

The whole receiver fits in one file. This version signs deliveries the way Stripe does (t=<timestamp>,v1=<hmac> over timestamp.payload), then runs them through verify, dedupe, order, apply. Run it with node receiver.mjs, no dependencies:

// a webhook receiver that survives duplicates and reordering
import { createHmac, timingSafeEqual } from "node:crypto";

const SECRET = "whsec_demo";
const TOLERANCE_SEC = 300; // reject signatures older than this

// sender side: every delivery is signed like Stripe: t=...,v1=...
function signed(evt, ts) {
  const payload = JSON.stringify(evt);
  const v1 = createHmac("sha256", SECRET)
    .update(`${ts}.${payload}`).digest("hex");
  return { payload, header: `t=${ts},v1=${v1}` };
}

// receiver side
const seen = new Set();              // event ids processed (persist this)
const state = new Map();             // object id -> { version, value }
const queue = [];

function receive({ payload, header }, nowSec) {
  const kv = Object.fromEntries(header.split(",").map(p => p.split("=", 2)));
  if (!kv.t || !kv.v1) return "400 malformed signature header";
  if (Math.abs(nowSec - Number(kv.t)) > TOLERANCE_SEC)
    return "400 stale timestamp, rejected";
  const expect = createHmac("sha256", SECRET)
    .update(`${kv.t}.${payload}`).digest("hex");
  if (kv.v1.length !== expect.length ||
      !timingSafeEqual(Buffer.from(kv.v1), Buffer.from(expect)))
    return "400 bad signature, rejected";
  const evt = JSON.parse(payload);
  if (seen.has(evt.id))
    return `200 ${evt.id} is a duplicate, acked and dropped`;
  seen.add(evt.id);
  queue.push(evt);                   // ack fast, work async
  return `200 ${evt.id} accepted`;
}

function processQueue() {
  for (const evt of queue.splice(0)) {
    const cur = state.get(evt.object);
    if (!cur || evt.version > cur.version) {
      state.set(evt.object, { version: evt.version, value: evt.value });
      console.log(`applied ${evt.id}: ${evt.object} v${evt.version} = ${evt.value}`);
    } else {
      console.log(`reorder: ${evt.id} (v${evt.version}) arrived after v${cur.version}, ignored`);
    }
  }
}

// the wire: sent in order, arrives scrambled + duplicated + hostile
const now = Math.floor(Date.now() / 1000);
const sent = [
  { id: "evt_01", object: "order_42", version: 100, value: "paid" },
  { id: "evt_02", object: "order_42", version: 110, value: "shipped" },
  { id: "evt_03", object: "order_42", version: 120, value: "refunded" },
];
const wire = [
  signed(sent[1], now),                        // evt_02 first
  signed(sent[2], now),                        // evt_03 next
  signed(sent[1], now),                        // evt_02 delivered again
  signed(sent[0], now),                        // evt_01 last
  { payload: JSON.stringify(sent[0]), header: `t=${now},v1=forged` },
  signed(sent[0], now - 600),                  // valid sig, 10 min stale
];

console.log("-- deliveries on the wire");
for (const d of wire) console.log(receive(d, now));
console.log("-- processing");
processQueue();
console.log("final:", state.get("order_42"));

Running it as-is produces this output:

-- deliveries on the wire
200 evt_02 accepted
200 evt_03 accepted
200 evt_02 is a duplicate, acked and dropped
200 evt_01 accepted
400 bad signature, rejected
400 stale timestamp, rejected
-- processing
applied evt_02: order_42 v110 = shipped
applied evt_03: order_42 v120 = refunded
reorder: evt_01 (v100) arrived after v120, ignored
final: { version: 120, value: 'refunded' }

Three details do the work. The signature covers the timestamp too, so an attacker cannot replay an old captured delivery with a fresh timestamp. The seen set answers "have I processed this exact delivery" with the id the vendor already gives you (Stripe's event id, Shopify's X-Shopify-Webhook-Id). And state is applied by the object's own version, so paid → shipped → refunded lands correctly no matter which order the wire picked.

Where it bites

The pattern has a floor you cannot skip and edges it does not cover.

Dedup is only as durable as the seen set. In-memory dies on redeploy, and Stripe will still be retrying: its live-mode window is three days, and a dashboard resend works up to 15 days after the event. The id set belongs in Postgres or Redis with a TTL at least as long as the vendor's retry window. Shopify splits the problem further: X-Shopify-Webhook-Id is unique per delivery while X-Shopify-Event-Id groups deliveries from the same merchant action, so the key you dedupe on depends on which duplicate you are killing.

Dedup does not make your handler idempotent. Stripe warns that "two separate Event objects are generated and sent" for some actions, which sail straight through an event-id check. The second layer is making the state change itself safe to run twice: upserts keyed on object id, version checks like the one above, and idempotency keys on any API call you make downstream (Stripe's own API "supports idempotency for safely retrying requests," keyed up to 24 hours).

Versions cannot order across topics. The version check works when events mutate one object. It does nothing for products/update arriving before products/create, or Stripe's own example where a subscription creation fires customer.subscription.created, invoice.created, invoice.paid, charge.created in no promised order. Handlers have to tolerate references that do not exist yet: store what arrived, upsert, reconcile later.

A surviving receiver can still lose the subscription. Shopify retries "8 times over the next 4 hours," then "the subscription is automatically deleted if it was configured using the Admin API." The deliveries simply stop; the only warning is an email to the developer address, and the endpoint goes quiet with no error to log. The safety net is the reconciliation job the pattern was supposed to replace. Keep it.

The numbers

• Stripe retries live-mode deliveries for up to three days with exponential backoff; sandbox deliveries get three retries over a few hours.
• Stripe's signature tolerance defaults to 5 minutes and the docs warn a value of 0 "disables the recency check entirely."
• Manual resends reach further than automatic retries: 15 days from the dashboard, 30 via the CLI, and the events API lists 30 days of history for reconciliation.
• Shopify gives you five seconds total, retries 8 times over 4 hours, then deletes Admin-API-configured subscriptions. Rolled secrets can take up to an hour to take over HMAC generation; Stripe keeps the previous secret signing for up to 24 hours.

Our position

Ship the queue-first receiver and nothing else: verify before you parse, dedupe before you process, ack before you work. Treat the reconciliation job as part of the same feature, not a later optimization, because the vendor's failure mode is silence, not an error you can log.

The takeaway

This is the pattern behind the integrations in our product engineering work, including the storefront sync we built for Grain and Vine. If your integration assumes one delivery in order, it is already dropping events; send us the endpoint and we will show you where.

← the journal