Crash Safety

OqronKit guarantees that no job is ever lost — even if your process is killed mid-execution. The guaranteedWorker flag is the foundation of this guarantee and is available across every module in OqronKit.

`guaranteedWorker` — Universal Guarantee

The guaranteedWorker option enables heartbeat-based crash-safe execution. When enabled, a HeartbeatWorker atomically claims each job and continuously renews a distributed lock while the handler runs. If the process dies, the lock expires and the job is automatically recovered.

Available in Every Module

Module	Option	Default
`Queue()`	`guaranteedWorker`	`true` — enabled unless explicitly set to `false`
`Worker()`	`guaranteedWorker`	`true` — enabled unless explicitly set to `false`
`Cron()`	`guaranteedWorker`	`false` — opt-in for critical crons
`Schedule()`	`guaranteedWorker`	`false` — opt-in for critical schedules
`Webhook()`	`guaranteedWorker`	`true` — delivery guarantees

Usage Across Modules

import { Queue, Worker, Cron, Schedule, Webhook } from "oqronkit";

// Queue — enabled by default, disable for fast fire-and-forget tasks
const fastQueue = Queue({
  name: "analytics-events",
  guaranteedWorker: false, // No heartbeat needed for lightweight tasks
  handler: async (ctx) => {
    /* ... */
  },
});

// Queue — default is true, critical financial processing
const billingQueue = Queue({
  name: "billing",
  heartbeatMs: 3_000, // Renew lock every 3s
  lockTtlMs: 15_000, // Lock expires after 15s if heartbeat stops
  handler: async (ctx) => {
    /* ... */
  },
});

// Worker — default is true, heavy compute
const videoWorker = Worker({
  topic: "video-encode",
  guaranteedWorker: true,
  heartbeatMs: 5_000,
  lockTtlMs: 30_000,
  handler: async (ctx) => {
    /* ... */
  },
});

// Cron — opt-in for critical scheduled jobs
const billingCron = Cron({
  name: "monthly-billing",
  expression: "0 0 1 * *",
  guaranteedWorker: true, // Critical — must survive crashes
  heartbeatMs: 5_000,
  lockTtlMs: 20_000,
  handler: async (ctx) => {
    /* ... */
  },
});

// Schedule — opt-in for critical one-off tasks
const migration = Schedule({
  name: "data-migration",
  runAt: new Date("2026-12-01"),
  guaranteedWorker: true, // Must complete — no lost migrations
  handler: async (ctx) => {
    /* ... */
  },
});

How It Works

Worker atomically claims a job → writes workerId + TTL to the Lock adapter
A heartbeat loop renews the lock every heartbeatMs while processing
If the process crashes (SIGKILL / OOM), the heartbeat stops
The lock expires in Redis/Postgres after lockTtlMs
The internal StallDetector finds the expired lock → marks the run as stalled
The job is re-queued and routed to a healthy worker within ~15 seconds

┌─ Worker Node A ────────────────────────────────────────┐
│  1. Claim job → lock(key, workerId, ttl=30s)          │
│  2. Start heartbeat → renew lock every 5s              │
│  3. Execute handler...                                 │
│  💥 CRASH (SIGKILL / OOM)                              │
└────────────────────────────────────────────────────────┘
                        ⏱️ Lock expires after 30s
┌─ Stall Detector ──────────────────────────────────────┐
│  4. Scan for expired locks                            │
│  5. Mark job as stalled                               │
│  6. Re-queue to broker                                │
└───────────────────────────────────────────────────────┘
┌─ Worker Node B ────────────────────────────────────────┐
│  7. Claim re-queued job → execute handler              │
│  8. Job completes successfully ✅                      │
└────────────────────────────────────────────────────────┘

Tuning Guide

Scenario	heartbeatMs	lockTtlMs	Why
Fast tasks (< 30s)	`5000`	`30000`	Standard protection
Heavy compute (minutes)	`10000`	`60000`	Longer TTL prevents premature stall
Critical financial	`3000`	`15000`	Aggressive detection, fast recovery
Spot instances	`5000`	`20000`	AWS/GCP can kill instances anytime

Rule of thumb: lockTtlMs should be at least 3 × heartbeatMs to tolerate temporary network delays.

Graceful Shutdown

When SIGINT or SIGTERM is received, OqronKit automatically captures it and triggers a graceful stop sequence:

Stops accepting new jobs from all modules (queues pause, schedulers stop ticking)
Waits for active jobs to drain (up to a configurable timeout)
Releases all held locks
Shuts down and closes all persistence adapter connections

You can also trigger this manually:

await oqron.stop();

AbortController Support

Handlers receive an AbortSignal via ctx.signal (and shorthand ctx.aborted). This signal is automatically aborted during a graceful shutdown sequence.

Handlers should check ctx.signal.aborted periodically or pass it down to outbound HTTP and database clients:

handler: async (ctx) => {
  for (const chunk of chunks) {
    if (ctx.signal.aborted) {
      throw new Error("Cancelled due to shutdown");
    }
    await processChunk(chunk);
    ctx.progress((i / chunks.length) * 100);
  }
};

Idempotency

Handlers will run more than once during crash scenarios. Use jobId as an idempotency key:

await emailQueue.add(
  { to: "user@example.com" },
  { jobId: "welcome-user@example.com" }, // Prevents duplicate processing
);

Disabled Behavior Engine

All modules support disabledBehavior for controlled pausing:

Behavior	When Disabled	Best For
`'hold'`	Accepts jobs in `paused` state, resumes on re-enable	Billing, order processing
`'skip'`	Silently drops jobs	Cache purges, analytics
`'reject'`	Throws error on `.add()`	API rate limiting feedback

The At-Least-Once Pipeline

Queues, workers, and webhooks share one crash-safety pipeline. Each mechanism closes a specific failure window:

Mechanism	Failure it closes
Durable-first writes — the job is persisted before the broker sees it	Producer crashes right after `.add()` returns
Atomic claims with TTL — one consumer owns a job at a time	Two nodes processing the same job
Heartbeat claim extension (`guaranteedWorker`)	Long jobs losing their claim mid-run
Owner-fenced ack/nack — a stale owner can't ack a re-claimed job	A partitioned node deleting work another node picked up
Boot reconciler — re-enqueues orphaned durable jobs on startup	Node dies between persist and process
Dead-letter queue — retries exhausted → `queue_dlq:<name>`, resendable	Poison messages retrying forever

add() ──► durable write ──► broker enqueue
                                 │
   poll ──► atomic claim (TTL) ──► run ──► heartbeat extends claim
                                    │
                        ok ──► fenced ack ──► retention
                        err ──► retry w/ backoff … exhausted ──► DLQ

Because delivery is at-least-once, a crash after side effects but before ack re-delivers the job — handlers should be idempotent (use idempotencyKey on producers and idempotent writes in handlers).

Scheduled work

Cron/Schedule handlers default to in-process, at-most-once execution — a crash mid-handler loses that fire (the schedule and history survive). Opt into the full pipeline per definition with durable: true: each fire becomes a deduplicated, slot-keyed job with retries, reclaim, and dead-lettering.

Next Steps

Architecture — runtime, modules, and the two-plane design
Adapters — backends, capabilities, and storage modes
Storage Model — where jobs, runs, and dead letters live

Crash Safety

On this page