Durable fan-out event delivery with HMAC signing, circuit breakers, per-endpoint limits, and resend

Webhook

Working example → Fan-out with HMAC signing, glob matching, per-endpoint rate limits, runtime endpoint CRUD, and resend: apps/backend/src/triggers/webhooks.ts

Webhook() defines an outbound event dispatcher. One .fire() fans out to every subscribed endpoint as an independent, durable delivery job — with its own retries, circuit breaker, rate limit, run history, and dead-letter entry. Deliveries ride the same at-least-once pipeline as queues: they survive crashes, restarts, and node failures.

Quick start

hooks/orders.ts

import { Webhook } from "oqronkit";

interface OrderEvent {
  orderId: string;
  amount: number;
}

export const orderHooks = Webhook<OrderEvent>({
  name: "order-hooks",
  timeout: 10_000,
  retries: { max: 5, strategy: "exponential", baseDelay: 5_000 }, // 5s, 10s, 20s…
  deadLetter: { enabled: true },
  security: { signingSecret: process.env.WEBHOOK_SECRET! },

  endpoints: [
    {
      name: "billing",
      url: "https://billing.internal/hooks",
      events: ["order.*"], // glob subscription
      rateLimit: { max: 100, windowMs: 60_000 },
    },
    {
      name: "analytics",
      url: "https://analytics.internal/ingest",
      events: ["*"], // everything
    },
  ],
});

routes/checkout.ts

// One fire → one durable delivery job per matching endpoint
const deliveries = await orderHooks.fire("order.completed", {
  orderId: order.id,
  amount: order.total,
});

// Or target one endpoint directly (no event matching)
await orderHooks.fireToEndpoint("billing", {
  orderId: order.id,
  amount: order.total,
});

Event matching

Endpoints subscribe with patterns; .fire(event, …) delivers to every enabled endpoint whose patterns match:

Pattern	Matches	Doesn't match
`order.completed`	exactly `order.completed`	anything else
`order.*`	`order.completed`, `order.line.added`, and bare `order`	`orders.created`, `user.created`
`*`	every event	—

Only exact names, the global *, and trailing prefix.* globs are supported. Mid-segment wildcards (user.*.activated) and ** are not.

Fan-out is per endpoint:

fire('order.completed', {...})
  ├─ ✓ billing    (order.*)  → durable job #1
  ├─ ✓ analytics  (*)        → durable job #2
  └─ ✗ user-svc   (user.*)   → no job

With an explicit idempotencyKey, each endpoint still gets its own deduplicated job (key:endpointName) — refiring the same event never double-delivers, and never collapses the fan-out:

await orderHooks.fire("order.completed", data, {
  idempotencyKey: `order-${order.id}`,
});

Delivery lifecycle

fire()
  ├─ match endpoints ─► durable job per endpoint ─► broker
  │
  │   poll ─► atomic claim ─► resolve endpoint (code ∪ runtime registry)
  │             │
  │             ├─ outbound rate limit blocked? ─► soft re-queue (no retry burned)
  │             ├─ circuit breaker OPEN?        ─► soft re-queue (no retry burned)
  │             │
  │             ├─ sign body (HMAC) + set headers
  │             ├─ HTTP request (timeout + abort)
  │             │
  │             ├─ 2xx           ─► completed · circuit success · onSuccess hook
  │             ├─ retryable     ─► retry with backoff (honors Retry-After) · circuit failure
  │             │   (408, 429, 500, 502, 503, 504 — configurable via retryOnStatus)
  │             └─ non-retryable ─► dead-letter immediately (401, 404, …) · onFail hook
  │
  └─ every attempt = a run (logs, timeline, duration, response status)

Two behaviors worth calling out:

Soft re-queues don't burn retries. A closed circuit or a rate-limited endpoint re-queues the delivery without consuming a retry attempt — transient endpoint pressure can't push deliveries into the DLQ.
Retry-After is honored. A 429/503 with Retry-After schedules the retry for exactly then (capped by retries.maxDelay), instead of blind backoff.

HMAC signing & receiver verification

With security configured, every delivery carries a signature over "{timestamp}.{body}":

Header	Content
`X-Oqron-Signature`	hex HMAC (sha256 default, sha512 optional)
`X-Oqron-Timestamp`	epoch ms used in the signature (replay protection)
`Idempotency-Key`	stable per delivery job — dedupe on your receiver

sender

Webhook({
  name: "order-hooks",
  security: {
    signingSecret: process.env.WEBHOOK_SECRET!,
    signingAlgorithm: "sha256", // or 'sha512'
    signingHeader: "X-Oqron-Signature",
    timestampHeader: "X-Oqron-Timestamp",
    // Or replace the built-in entirely:
    // signFunction: (body, secret, ts) => myCustomSignature(body, secret, ts),
  },
  endpoints: [/* ... */],
});

receiver (your endpoint)

import { createHmac, timingSafeEqual } from "node:crypto";

app.post("/hooks", (req, res) => {
  const ts = req.header("X-Oqron-Timestamp")!;
  const sig = req.header("X-Oqron-Signature")!;

  if (Date.now() - Number(ts) > 5 * 60_000) return res.status(401).end(); // replay window

  const expected = createHmac("sha256", process.env.WEBHOOK_SECRET!)
    .update(`${ts}.${req.rawBody}`)
    .digest("hex");

  if (!timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
    return res.status(401).end();
  }
  // process… respond 2xx fast; do heavy work async
  res.status(200).end();
});

security can also be an async function (per-delivery secret resolution), and each endpoint can override it.

Circuit breaker

Each dispatcher:endpoint pair has an independent circuit — a failing endpoint stops receiving traffic instead of burning retries:

5 consecutive failures → circuit opens; deliveries soft re-queue.
After 30s → half-open; one trial delivery goes through.
Trial succeeds → closed. Trial fails → open again.

State is stored durably, so all nodes share one view of an endpoint's health, and breaker bookkeeping fails open — a broken state store never blocks deliveries.

Dynamic endpoints & runtime registry

Endpoints can come from code (static array or resolver function) and from a persisted runtime registry — merged at delivery time, registry wins:

// Code-side resolver (e.g. from your subscriptions table)
Webhook<OrderEvent>({
  name: "order-hooks",
  endpoints: async () => {
    const subs = await WebhookSubscriptions.findAll({
      where: { active: true },
    });
    return subs.map((s) => ({
      name: s.client,
      url: s.targetUrl,
      events: s.events,
    }));
  },
});

// Runtime CRUD (persisted, effective without redeploy)
await orderHooks.addEndpoint({
  name: "partner-x",
  url: "https://x.com/hooks",
  events: ["order.*"],
});
await orderHooks.disableEndpoint("partner-x"); // stops fan-out, keeps config
await orderHooks.enableEndpoint("partner-x");
await orderHooks.removeEndpoint("partner-x");
await orderHooks.getEndpoints(); // merged view

Per-endpoint overrides: method, headers (object or function of the payload), url as a function of the payload, security, retries.retryOnStatus, rateLimit.

Dead letters & resend

Retries exhausted (or a non-retryable status) → the delivery lands in queue_dlq:<dispatcher> with its full payload:

// Inspect
const dead = await oqron.persistence.records("queue_dlq:order-hooks").list();

// Re-deliver — clones the payload as a fresh job (new idempotency key)
const newJobId = await orderHooks.resend(dead[0].id);

// Get notified
Webhook({
  name: "order-hooks",
  deadLetter: { enabled: true, onDead: (job) => alert(job) },
});

Hooks, throughput & control

Webhook<OrderEvent>({
  name: "order-hooks",
  concurrency: 10, // parallel deliveries per node
  throttle: { max: 50, duration: 60_000 }, // per-node dispatch cap
  hooks: {
    onSuccess: (job, result) => metrics.timing("webhook.ms", result.durationMs),
    onFail: (job, error) => {}, // fires per failed attempt
  },
  transform: (data, endpoint) => ({ ...data, deliveredTo: endpoint.name }),
  environments: ["production"], // inert elsewhere
  endpoints: [/* ... */],
});

await orderHooks.pause(); // stop claiming deliveries (jobs keep accumulating durably)
await orderHooks.resume();

Reliability semantics

Situation	Outcome
Node crashes mid-delivery	Claim TTL expires → another node retries (at-least-once)
Node crashes after `fire()`	Job was durable-first → boot reconciler re-enqueues
Endpoint down	Retries with backoff → circuit opens → soft re-queues → recovers when endpoint does
Endpoint returns 429 + Retry-After	Retry scheduled for exactly that time
Endpoint removed/disabled mid-flight	Delivery dead-letters (with `onDead`)
Same event fired twice with `idempotencyKey`	Deduplicated per endpoint — one delivery each

Every attempt is recorded as a run (module: 'webhook') with HTTP status, duration, and timeline.

Next Steps

Observability & Control — delivery history, metrics, DLQ management
Crash Safety — the at-least-once pipeline deliveries ride on
Rate Limiter — cluster-wide limits beyond per-endpoint caps

Webhook

On this page