Reliability and recovery

Pulse uses MongoDB documents, unique indexes, renewable leases, and reconciliation to provide recoverable delivery without requiring multi-document transactions.

Delivery lifecycle

For every event and consumer group, Pulse follows this lifecycle:

event
  → durable delivery (pending)
  → history attempt (pending, no lock)
  → history attempt (pending, leased)
  → handler
  → history attempt (success or error)
  → delivery (success, retry, or terminal error)

The history attempt exists before the handler starts. Operators can therefore distinguish work that is queued, currently running, or abandoned by a dead worker.

History state	Meaning
`pending` without `lockedUntil`	Queued or waiting for `nextAttemptAt`.
`pending` with a future `lockedUntil`	A worker is executing it.
`pending` with an expired `lockedUntil`	The worker or machine stopped renewing its lease.
`success`	The handler completed and Pulse acknowledged the attempt.
`error`	The handler failed, the worker was lost, or the payload expired.

Leases, heartbeats, and fencing

When a worker acquires an attempt, it writes:

startedAt, lockedAt, and lockedUntil
a UUIDv7 lockOwner
a new UUIDv7 lockToken
heartbeatAt

Pulse renews the attempt lease approximately every third of lockTimeoutMs. Ordered consumers also renew the topic ordering lease. Every success or error update includes the fencing token in its MongoDB filter. If another replica has already recovered the attempt, the stale worker’s token no longer matches and it cannot overwrite the new result. Choose a lockTimeoutMs that comfortably exceeds expected event-loop stalls and temporary MongoDB connectivity interruptions. Heartbeats keep normal long-running asynchronous handlers alive, but blocking the JavaScript event loop can prevent renewal.

Recovery after a crash

Every worker loop runs three recovery steps before looking for new events:

Reap one expired attempt.
Reconcile incomplete delivery state.
Acquire existing work or discover new events.

An expired attempt is atomically changed from pending to error with code worker_lost. At-least-once mode then creates the next numbered attempt; at-most-once mode makes the delivery terminal. Reconciliation repairs safe partial states idempotently:

Delivery exists but its first history attempt does not.
History is successful but delivery is still pending.
History is errored but its retry was not created.
Event discovery was interrupted before the subscription cursor advanced.

Unique indexes on consumerGroup + eventId and deliveryId + attempt ensure replicas converge on one logical record.

Crash windows

Crash point	Recovery	Handler may run again?
Before the handler starts	Expired lease becomes `worker_lost`; retry is created.	Yes, but the previous handler did not start.
During the handler	Expired lease becomes `worker_lost`; retry is created.	Yes.
After a side effect but before history success	Another attempt is created.	Yes.
After history success but before delivery success	Reconciliation finishes the delivery from successful history.	No.
After history error but before retry creation	Reconciliation creates exactly the next numbered attempt.	Yes.

At-least-once delivery cannot prove whether an external side effect happened before a machine disappeared. Exactly-once side effects require idempotency in the system receiving the side effect.

Design idempotent handlers

Use event.id as the idempotency key sent to an external API:

await pulse.subscribe('payment.captured', async event => {
  await ledger.recordPayment({
    idempotencyKey: event.id,
    paymentId: event.data.paymentId,
    orderId: event.data.orderId,
  })
})

When the side effect is in MongoDB, store the event ID with a unique index in the same transaction as the domain write:

await mongoClient.withSession(async session => {
  await session.withTransaction(async () => {
    await processedEvents.insertOne(
      {_id: event.id, processedAt: new Date()},
      {session},
    )
    await orders.updateOne(
      {_id: event.data.orderId},
      {$set: {invoiced: true}},
      {session},
    )
  })
})

Treat a duplicate-key error on processedEvents._id as an already completed handler. Do not acknowledge the idempotency marker before a non-transactional side effect that can still fail.

Error codes

Pulse writes these built-in error codes to history:

Code	Meaning
`handler_error`	The handler threw or rejected.
`worker_lost`	The attempt lease expired before completion.
`event_expired`	Retention removed the event payload before execution.

Internal errors and lost-fencing notifications are delivered to onError. The callback should report or log quickly and must not throw:

const pulse = connect<AppEvents>({
  connectionString,
  databaseName: 'commerce',
  consumerGroup: 'billing',
  onError(error) {
    logger.error('Pulse worker error', {error})
  },
})

Polling and Change Streams

Polling and reconciliation guarantee progress. Change Streams only wake sleeping workers after new event inserts.

`changeStreams`	Behavior
`auto`	Enable on replica sets or sharded clusters; fall back to polling on standalone MongoDB. Default.
`required`	Fail `awaitConnection()` unless MongoDB supports Change Streams.
`disabled`	Use polling only.

If an active Change Stream fails, Pulse reports the error and attempts to open it again after pollIntervalMs. Missing a notification does not lose the event.

Collections and retention

With the default prefix, Pulse owns:

Collection	Retention
`orionjs.pulse.events`	`eventRetentionMs` from publication time.
`orionjs.pulse.subscriptions`	Durable; no TTL.
`orionjs.pulse.deliveries`	`historyRetentionMs` after terminal completion.
`orionjs.pulse.history`	`historyRetentionMs` after the attempt ends.

Retention uses absolute expiresAt dates and TTL indexes with expireAfterSeconds: 0. When retention is null, Pulse omits expiresAt; MongoDB keeps those documents even though the TTL index remains.

Startup safety

During initialization, Pulse creates missing collections and named indexes, then inspects their keys and relevant options. Startup fails before workers begin when:

MongoDB permissions do not allow the required setup.
Existing data violates a required unique index.
A named Pulse index has incompatible keys, uniqueness, or TTL configuration.
changeStreams: 'required' is used on an unsupported topology.

Pulse never drops or replaces an incompatible index automatically. The error identifies the collection, index name, expected configuration, and actual configuration so an operator can make the migration explicitly. See the API reference for the exact index names and configuration defaults.

Get Started

Essentials

Controllers

Other Modules

Reliability and recovery

Delivery lifecycle

Leases, heartbeats, and fencing

Recovery after a crash

Crash windows

Design idempotent handlers

Error codes

Polling and Change Streams

Collections and retention

Startup safety

​Delivery lifecycle

​Leases, heartbeats, and fencing

​Recovery after a crash

​Crash windows

​Design idempotent handlers

​Error codes

​Polling and Change Streams

​Collections and retention

​Startup safety

Delivery lifecycle

Leases, heartbeats, and fencing

Recovery after a crash

Crash windows

Design idempotent handlers

Error codes

Polling and Change Streams

Collections and retention

Startup safety