Reliability and recovery

Pulse combines durable MongoDB state, expiring leases, fencing tokens, and retries to provide recoverable concurrent delivery. Queue state and attempt outcomes live on the delivery document, so claiming, retrying, and completing an attempt do not require cross-collection transactions. Batch materialization uses one short transaction to insert its delivery and advance the discovery cursor.

Delivery lifecycle

Normal receivers produce one delivery per event and consumer group. Batch receivers group up to batchSize ordered events into one delivery:

event discovered
  → one event or an ordered batch is queued
  → attempt claimed with a lock token
  → handler runs while the lease is renewed
  → success, retry, or terminal error committed atomically
  → retention applied

Pulse persists the claim before invoking the handler. A delivery stores:

Its queue or terminal state.
The current attempt number and retry deadline.
The active lock owner, token, and expiration.
The terminal outcome and error when present.
A bounded window containing the latest ten completed attempts.

Delivery guarantees

At-least-once

This is the default. If a process disappears before acknowledging an attempt, another replica can recover it after the lease expires. A handler may therefore run more than once.

At-most-once attempt

With delivery: 'at-most-once', Pulse does not retry after a handler starts. This reduces duplicate execution but can lose processing if the worker disappears mid-handler. Neither mode can make external side effects exactly-once. Use event.id as an idempotency key.

Leases, heartbeats, and fencing

An active attempt stores a random lockToken and lockedUntil. Pulse renews the lease about every third of lockTimeoutMs while the callback runs. Every completion update matches both the delivery ID and lock token. If the lease was reaped and a new worker owns the delivery, a stale worker cannot acknowledge or reschedule it. Pulse reports PulseLockLostError instead. Choose lockTimeoutMs comfortably above ordinary MongoDB latency. Heartbeats protect handlers that run longer than the timeout; they do not require handlers to finish within it.

Crash recovery

Only the discovery leader for a topic performs maintenance:

Scan a bounded indexed batch of expired delivery locks.
Commit the expired attempt as an error.
Schedule the next retry or mark the delivery terminal.
Periodically delete successful deliveries whose events are behind the persisted cursor.

Expired-lock reaping and delivery cleanup use their own schedules outside the handler hot loop. When a bounded batch is full, Pulse immediately continues draining; otherwise it waits before the next maintenance pass. Because each attempt transition is one atomic delivery update, there is no attempt/delivery reconciliation phase.

Discovery cursors

New events receive a MongoDB BSON timestamp through $currentDate. Pulse uses this server-assigned sequence as its durable discovery order, so publishers with skewed application clocks cannot leave events behind an advanced cursor. The discovery leader advances a normal receiver cursor only after materializing its deliveries. For a batch receiver, inserting the multi-event delivery and advancing the cursor to the last event are committed in one MongoDB transaction. A crash therefore commits both changes or neither, preventing overlapping batches and skipped events.

Failure matrix

Failure point	Durable result	Handler may repeat?
Before delivery materialization	Discovery cursor is not advanced, so the event is discovered again.	No, unless an earlier attempt already ran.
After materialization but before claim	Another worker can claim the queued delivery.	No.
During the handler	The lease expires and the attempt is retried according to policy.	Yes.
After the side effect but before acknowledgement	The lease expires and the attempt can run again.	Yes.
During acknowledgement	The atomic update either commits completely or loses its fencing-token match.	According to committed state.
After terminal state but before TTL deletion	The terminal delivery remains queryable until cleanup or TTL removes it.	No.

Idempotent handlers

Use the event ID as the unique key for durable side effects:

await pulse.subscribe('payment.captured', async event => {
  await db.collection('ledgerEntries').updateOne(
    {eventId: event.id},
    {
      $setOnInsert: {
        eventId: event.id,
        paymentId: event.data.paymentId,
        amount: event.data.amount,
        createdAt: new Date(),
      },
    },
    {upsert: true},
  )
})

Enforce a unique index on the idempotency key:

db.ledgerEntries.createIndex({eventId: 1}, {unique: true})

For remote APIs, send event.id as their idempotency key when supported. If a workflow spans multiple writes, use a local transaction or an outbox/inbox design.

Retry policy

The delay after failed attempt n is:

retryDelayMs × retryBackoffMultiplier ^ (n - 1)

maxRetries counts retries after the first attempt. Pulse retains the latest ten completed attempt outcomes inside the delivery document until that delivery expires.

Retention

Data	Retention
Events	`eventRetentionMs` after publication.
Terminal deliveries and their attempt history	`historyRetentionMs` after completion.

Set either value to null to disable that TTL. MongoDB TTL deletion is asynchronous. The discovery leader also deletes successful deliveries in bounded batches, but only after its persisted cursor has passed the event. When historyRetentionMs is configured, this cleanup requires expiresAt so it cannot bypass retention. With historyRetentionMs: null, the cursor alone is sufficient.

Operational guidance

Keep all replicas of one consumer group on matching persisted settings.
Increase configVersion whenever those settings change.
Make handlers idempotent before increasing concurrency.
Monitor terminal errors, retry age, expired locks, and delivery backlog.
Keep event retention longer than the maximum expected outage or retry window.
Use a new consumer group when an independent replay of retained events is required.

Get Started

Essentials

Controllers

Other Modules

Reliability and recovery

Delivery lifecycle

Delivery guarantees

At-least-once

At-most-once attempt

Leases, heartbeats, and fencing

Crash recovery

Discovery cursors

Failure matrix

Idempotent handlers

Retry policy

Retention

Operational guidance

​Delivery lifecycle

​Delivery guarantees

​At-least-once

​At-most-once attempt

​Leases, heartbeats, and fencing

​Crash recovery

​Discovery cursors

​Failure matrix

​Idempotent handlers

​Retry policy

​Retention

​Operational guidance

Delivery lifecycle

Delivery guarantees

At-least-once

At-most-once attempt

Leases, heartbeats, and fencing

Crash recovery

Discovery cursors

Failure matrix

Idempotent handlers

Retry policy

Retention

Operational guidance