Signed, retryable outgoing webhooks: HMAC, an outbox, and BullMQ

"Send a webhook" sounds like one line of code — await fetch(url, { method: 'POST', body }) — and it is, right up until it's a production feature other people build on. Then it's three separate problems wearing a trench coat:

1. Authenticity. The receiver has to know the payload came from you and wasn't replayed. That's a signature.
2. The fanout must not break the business write. Creating an invitation should not fail because a customer's webhook endpoint is down.
3. Delivery must survive a bad receiver. Timeouts, 500s, a receiver that's redeploying — every one of those needs a retry with backoff, not a dropped event.

UseDeploy splits those into three layers that each do one job: HMAC signing at the delivery boundary, a durable outbox for the domain-event fanout, and BullMQ for per-delivery retries. This post is the tour. The webhooks feature page has the product view; the event-bus how-to covers the bus your own code publishes to.

The path an event takes

A business write emits a domain event. That event lands in the outbox. A listener maps the internal event name to a public one and enqueues one BullMQ job per subscribed endpoint. The worker signs and POSTs. If the receiver doesn't return 2xx, BullMQ retries. Concretely:

create invitation
  → publish "tenancy.invitation.created" (outbox row, then inline dispatch)
    → webhook listener maps it to public "invitation.created"
      → enqueue one BullMQ job per subscribed, enabled endpoint
        → worker: sign (HMAC) + POST with a 10s timeout
          → non-2xx? throw → BullMQ schedules a retry

Two retry mechanisms live on this path, and it's worth being precise about which is which. The outbox makes the domain-event fanout durable (at-least-once, up to 8 attempts). BullMQ makes each individual HTTP delivery durable (3 attempts, exponential backoff). They're different layers solving different failures; conflating them is how you end up double-counting retries.

Layer 1: the signature

Signatures are generated in the domain, and the format is deliberately Stripe-shaped so any receiver that already integrates Stripe knows the drill:

export const buildSignatureHeader = (
  secret: string,
  rawBody: string,
  timestampSeconds: number,
): string => {
  const signingInput = `${timestampSeconds}.${rawBody}`;
  const v1 = createHmac('sha256', secret).update(signingInput).digest('hex');
  return `t=${timestampSeconds},v1=${v1}`;
};

The header is t=<unix>,v1=<hmac-sha256-hex>, and the signed bytes are ${timestamp}.${rawBody} — not the body alone. Signing the timestamp with the body is what makes the signature replay-resistant: a receiver rejects a request whose t is too old, and an attacker can't lift a valid signature onto a stale payload because the timestamp is inside the HMAC.

The secret itself is high-entropy and prefixed so it survives a trip through an env var or JSON without escaping:

export const generateWebhookSecret = (): string =>
  `whsec_${randomBytes(32).toString('hex')}`;

Receivers reconstruct ${t}.${rawBody}, recompute the HMAC with their copy of the secret, and compare with a constant-time equality check. Standard, boring, correct — which is exactly what you want in a signature scheme.

Layer 2: the outbox, so the fanout can't break the write

The event bus in production is a transactional-outbox-lite. publish() writes every event as a pending event_outbox row first, then attempts inline dispatch:

async publish(events: ReadonlyArray<DomainEvent>): Promise<void> {
  if (events.length === 0) return;
  const rows = events.map((e) => ({
    id: randomUUID(),
    name: e.name,
    aggregateId: e.aggregateId,
    // JSON round-trip so the FIRST delivery sees the exact same payload
    // shape a DB redelivery would (Dates → ISO strings, undefined dropped).
    payload: JSON.parse(JSON.stringify(e.payload)) as Record<string, unknown>,
    occurredAt: e.occurredAt,
  }));
  await this.prisma.eventOutbox.createMany({ data: rows.map(/* ... */) });
  for (const row of rows) await this.attemptDispatch(row);
}

The consequences are chosen carefully:

• An insert failure rejects publish. In the billing-webhook path that becomes a 5xx, so the provider redelivers. Fail loud by design.
• A handler failure does not reject. The row stays pending with a lastError, captureError pages someone, and a background poller retries with exponential backoff — 30s, 1m, 2m, 4m, 8m, capped at 15m — until success or dead-letter (status = failed) after 8 attempts.
• Delivery is at-least-once per event, not per handler. A retry re-runs every handler subscribed to that event, so handlers must be idempotent or tolerate the rare duplicate.

That JSON round-trip on the payload is a small detail with a big return: the very first inline dispatch sees the same serialized shape (Dates as ISO strings, undefined dropped) that a database redelivery would. Serialization bugs surface on attempt one instead of hiding until a retry three days later.

The poller is safe to run in more than one process because the claim is optimistic — it increments attempts only if the row is still pending at the attempt number it read:

const claimed = await this.prisma.eventOutbox.updateMany({
  where: { id: row.id, status: 'pending', attempts: row.attempts },
  data: { attempts: { increment: 1 } },
});
if (claimed.count !== 1) return; // someone else won this attempt

So the API process and the worker can both poll (redundancy) without ever double-dispatching the same attempt. The one residual gap is stated honestly in the code: a crash exactly between the aggregate save() and publish() still drops the event. Closing it needs a unit-of-work spanning both writes, which is deliberately out of scope for a boilerplate.

Layer 3: fanout and delivery

A listener subscribes to the internal domain events and translates them to the public catalog. Internal names are an implementation detail; the public names are the contract customers subscribe to:

const PUBLIC_EVENT_MAP = [
  { internal: 'tenancy.membership.added',        public: 'member.added' },
  { internal: 'tenancy.invitation.created',      public: 'invitation.created' },
  { internal: 'tenancy.organization.created',    public: 'organization.created' },
  { internal: 'billing.subscription.updated',    public: 'subscription.updated' },
  { internal: 'iam.user.registered',             public: 'user.registered' },
  // ...
];

The wiring swallows errors on purpose — the fanout must never break the originating business write:

bus.subscribe(internal, async (event) => {
  try {
    const payload = event.payload as Record<string, unknown>;
    const organizationId =
      (typeof payload.organizationId === 'string' ? payload.organizationId : null) ??
      (typeof payload.orgId === 'string' ? payload.orgId : null);
    if (!organizationId) return; // e.g. user.registered isn't org-scoped
    await enqueue.execute({ organizationId, eventType: publicName, payload: /* ... */ });
  } catch (err) {
    logger.warn({ err, event: event.name }, 'webhook enqueue failed');
  }
});

Enqueue looks up the org's enabled endpoints, filters to the ones subscribed to this event type, and schedules one BullMQ job per endpoint. Each job carries its own pre-built payload and endpoint id, so the worker is a pure HTTP poster with no DB lookups in the hot path:

const jobId = `whdeliver-${endpoint.id}-${input.eventType}-${Date.now()}`;
await this.scheduler.schedule(QUEUE_NAMES.webhooksOutgoing, 'deliver', data, {
  jobId,
  attempts: 3,
});

(That jobId uses - as a separator for a real reason: BullMQ rejects custom ids containing :. An earlier version used : and silently dropped every delivery — the kind of bug that only a test against a real Redis catches, which is why the webhook e2e path runs against real Redis rather than a mock.)

The delivery attempt itself

The worker's DeliverWebhookUseCase resolves the endpoint (using the audited system client, because a BullMQ job has no request org — the endpoint row is the tenant), builds the envelope, signs it, and POSTs:

const envelope = {
  id: `evt_${data.occurredAt}_${data.eventType}`,
  type: data.eventType,
  occurredAt: data.occurredAt,
  data: data.payload,
};
const rawBody = JSON.stringify(envelope);
const ts = Math.floor(this.clock().getTime() / 1000);
const signature = buildSignatureHeader(endpoint.secret, rawBody, ts);

const result = await this.httpPoster.post(endpoint.url, rawBody, {
  'Content-Type': 'application/json',
  'X-Webhook-Signature': signature,
  'X-Webhook-Event': data.eventType,
  'X-Webhook-Timestamp': String(ts),
});

Every attempt is recorded to a WebhookDelivery row — success or failure, HTTP status, attempt number, a hash of the payload — before the method returns its outcome. That ordering is intentional: the use case throws on non-2xx so BullMQ schedules a retry, and recording first means the audit trail is complete even when the throw kicks the worker into its retry path. The endpoint's lastDeliveryAt / lastDeliveryStatus get bumped in the same step so the dashboard shows the real state.

The HTTP poster is the last line of defense against a slow receiver. It uses global fetch with a hard 10-second AbortSignal timeout, and drains the response body to free the socket:

export class FetchHttpPoster implements IHttpPoster {
  constructor(private readonly timeoutMs = 10_000) {}

  async post(url: string, body: string, headers: Record<string, string>) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), this.timeoutMs);
    try {
      const res = await fetch(url, { method: 'POST', headers, body, signal: controller.signal });
      try { await res.text(); } catch { /* best-effort drain */ }
      return { status: res.status };
    } finally {
      clearTimeout(timer);
    }
  }
}

A receiver that hangs for a minute is a classic way to jam an outbound worker pool. The timeout turns "hang forever" into "fail in 10 seconds, retry later."

Then the processor decides retry-or-done based purely on the outcome:

const outcome = await deliver.execute(job.data, attempt);
if (outcome.status !== 'success') {
  logger.warn({ jobId: job.id, attempt, httpStatus: outcome.httpStatus }, 'webhook delivery failed');
  throw new Error(outcome.error ?? `delivery failed (status=${outcome.httpStatus ?? 'n/a'})`);
}

Throwing hands control back to BullMQ, which applies the exponential backoff configured at enqueue time and reschedules until the attempt budget is spent.

Why this shape

Each layer has exactly one failure it owns, and none of them can silently eat an event:

• The signature owns authenticity and replay — a receiver can verify you and reject stale payloads.
• The outbox owns "the fanout must not break the write" and "a slow handler must not lose the event" — it persists first, retries with backoff, dead-letters visibly, and pages on the way down.
• BullMQ owns per-delivery HTTP retries — a receiver being down for a minute costs a retry, not a dropped event.

The seam between them is the domain event. Your own use cases publish to the same bus (see the event-bus how-to), and the webhook fanout is just another subscriber — so exposing a new public webhook is a one-line addition to the event map, not a new delivery pipeline. If you're wiring outgoing webhooks into your own product, the webhooks feature page is the place to start; this post is what's running underneath the "add endpoint" button.