Reliability and recovery

Pulse uses MongoDB documents, unique indexes, renewable leases, and reconciliation to provide recoverable delivery without requiring multi-document transactions.

Delivery lifecycle

For every event and consumer group, Pulse follows this lifecycle:

event
  → durable delivery (pending)
  → history attempt (pending, no lock)
  → history attempt (pending, leased)
  → handler
  → history attempt (success or error)
  → delivery (success, retry, or terminal error)

The history attempt exists before the handler starts. Operators can therefore distinguish work that is queued, currently running, or abandoned by a dead worker.

History state	Meaning
`pending` without `lockedUntil`	Queued or waiting for `nextAttemptAt`.
`pending` with a future `lockedUntil`	A worker is executing it.
`pending` with an expired `lockedUntil`	The worker or machine stopped renewing its lease.
`success`	The handler completed and Pulse acknowledged the attempt.
`error`	The handler failed, the worker was lost, or the payload expired.

Leases, heartbeats, and fencing

When a worker acquires an attempt, it writes:

startedAt, lockedAt, and lockedUntil
a UUIDv7 lockOwner
a new UUIDv7 lockToken
heartbeatAt

Pulse renews the attempt lease approximately every third of lockTimeoutMs. Ordered consumers also renew the topic ordering lease. Every success or error update includes the fencing token in its MongoDB filter. If another replica has already recovered the attempt, the stale worker’s token no longer matches and it cannot overwrite the new result. Choose a lockTimeoutMs that comfortably exceeds expected event-loop stalls and temporary MongoDB connectivity interruptions. Heartbeats keep normal long-running asynchronous handlers alive, but blocking the JavaScript event loop can prevent renewal. Calling close() stops acquisition of new work but keeps heartbeats active for callbacks already running. Pulse waits for those callbacks to settle before closing MongoDB, preventing another replica from reaping intentionally draining work during graceful shutdown. A handler may initiate and await close() without deadlocking itself. Pulse starts the graceful close and lets that handler return so its attempt can be acknowledged. Lifecycle code outside the handler can call close() again to await the shared close promise.

Recovery after a crash

The process coordinator runs three recovery steps before looking for new events:

Reap a bounded batch of expired attempts.
Reconcile a bounded batch of incomplete delivery state.
Acquire existing work or discover new events.

An expired attempt is atomically changed from pending to error with code worker_lost. At-least-once mode then creates the next numbered attempt; at-most-once mode makes the delivery terminal. Reconciliation repairs safe partial states idempotently:

Delivery exists but its first history attempt does not.
History is successful but delivery is still pending.
History is errored but its retry was not created.
Event discovery was interrupted before the subscription cursor advanced.
A terminal delivery was written but its history and delivery TTL dates were not.

Unique indexes on consumerGroup + eventId and deliveryId + attempt ensure replicas converge on one logical record.

Concurrent publication ordering

Pulse assigns every new event an internal BSON timestamp in the same MongoDB write that creates the event. Discovery advances through that MongoDB-assigned sequence instead of relying on application clocks or the order in which concurrent publish() promises resolve. This prevents a slow concurrent insert from appearing behind an already-advanced cursor. The public createdAt field remains available as the application timestamp. Ordered consumers execute new events by the same MongoDB sequence; legacy events without a sequence fall back to createdAt + eventId. Events created by older Pulse versions do not have the internal sequence. Upgraded consumers keep an independent createdAt + _id cursor for those legacy events and check it before the MongoDB-assigned cursor, so old and new publishers can coexist during a rolling upgrade. Subscriptions first created before Pulse 4.4.3 may have only the legacy cursor. Their first current consumer safely scans the retained sequenced backlog and relies on the delivery unique key to avoid re-execution. Keep historyRetentionMs at least as long as eventRetentionMs until every durable subscription has a cursorSequence; otherwise an event whose old delivery record already expired can be delivered again, consistent with Pulse’s at-least-once guarantee.

Crash windows

Crash point	Recovery	Handler may run again?
Before the handler starts	Expired lease becomes `worker_lost`; retry is created.	Yes, but the previous handler did not start.
During the handler	Expired lease becomes `worker_lost`; retry is created.	Yes.
After a side effect but before history success	Another attempt is created.	Yes.
After history success but before delivery success	Reconciliation finishes the delivery from successful history.	No.
After history error but before retry creation	Reconciliation creates exactly the next numbered attempt.	Yes.

At-least-once delivery cannot prove whether an external side effect happened before a machine disappeared. Exactly-once side effects require idempotency in the system receiving the side effect.

Design idempotent handlers

Use event.id as the idempotency key sent to an external API:

await pulse.subscribe('payment.captured', async event => {
  await ledger.recordPayment({
    idempotencyKey: event.id,
    paymentId: event.data.paymentId,
    orderId: event.data.orderId,
  })
})

When the side effect is in MongoDB, store the event ID with a unique index in the same transaction as the domain write:

await mongoClient.withSession(async session => {
  await session.withTransaction(async () => {
    await processedEvents.insertOne(
      {_id: event.id, processedAt: new Date()},
      {session},
    )
    await orders.updateOne(
      {_id: event.data.orderId},
      {$set: {invoiced: true}},
      {session},
    )
  })
})

Treat a duplicate-key error on processedEvents._id as an already completed handler. Do not acknowledge the idempotency marker before a non-transactional side effect that can still fail.

Error codes

Pulse writes these built-in error codes to history:

Code	Meaning
`handler_error`	The handler threw or rejected.
`worker_lost`	The attempt lease expired before completion.
`event_expired`	Retention removed the event payload before execution.

Internal errors and lost-fencing notifications are delivered to onError. The callback should report or log quickly and must not throw:

const pulse = connect<AppEvents>({
  connectionString,
  databaseName: 'commerce',
  consumerGroup: 'billing',
  onError(error) {
    logger.error('Pulse worker error', {error})
  },
})

MongoDB connection capacity

Pulse configures maxPoolSize: 5 by default. With 500 service replicas, that bounds the application pool for the active MongoDB server to 2,500 connections instead of the driver’s default ceiling of 50,000. Pools start empty because minPoolSize remains 0 and grow only when concurrent operations need them. The limit is per MongoClient and per server in the MongoDB topology. The driver also keeps monitoring connections outside the application pool. If all five connections are busy, operations wait for one to return rather than opening beyond the configured maximum. Tune the limit per service when needed:

const pulse = connect({
  connectionString,
  databaseName: 'commerce',
  consumerGroup: 'high-throughput-worker',
  maxPoolSize: 10,
})

Keep the default for typical ordered consumers. Increase it only after observing pool contention or when using higher worker and handler concurrency.

Polling

Polling and reconciliation are Pulse’s only discovery, acquisition, and recovery mechanisms. Pulse does not open MongoDB Change Streams. Every process has one coordinator regardless of workerCount. That coordinator performs one acquisition query across every local topic per poll, instead of one query per topic. Candidate topics rotate in bounded batches, so a busy topic cannot permanently hide another one. Use pollIntervalMs to balance idle query volume against delivery latency.

Since Pulse 4.5.2, handler slots never poll MongoDB independently. workerCount only controls how many callbacks may execute concurrently; increasing it does not multiply the process’s discovery, acquisition, or recovery polling loops.

Discovery leadership uses one renewable lease per consumerGroup + topic, configured with discoveryLockTimeoutMs and defaulting to 10 seconds. A process acquires and renews all of its topic leases in bulk, then discovers its owned topics in one aggregate query. Lease and event polling thus scale with processes rather than processes multiplied by topics, while a replica that registers a different topic set can still own and recover those topics safely. Cursor updates are fenced by the same topic lease token. If a query finishes after its lease was lost, the stale reader may create idempotent delivery records but cannot advance the cursor. The new owner safely repeats discovery. Temporary query errors do not voluntarily abandon a lease; expiry still permits failover.

Collections and retention

With the default prefix, Pulse owns:

Collection	Retention
`orionjs.pulse.events`	`eventRetentionMs` from publication time.
`orionjs.pulse.subscriptions`	Durable; no TTL.
`orionjs.pulse.deliveries`	`historyRetentionMs` after terminal completion.
`orionjs.pulse.history`	`historyRetentionMs` after its delivery becomes terminal.

Retention uses absolute expiresAt dates and TTL indexes with expireAfterSeconds: 0. When retention is null, Pulse omits expiresAt; MongoDB keeps those documents even though the TTL index remains. Errored attempts that still have a retry do not receive an expiry date. This preserves the retry number and recovery evidence even when the retry delay or a complete service outage lasts longer than historyRetentionMs. Once the delivery becomes terminal, Pulse assigns the same terminal retention deadline to all its completed attempts, then makes the delivery expirable. If a machine stops between those writes, the non-expiring terminal delivery remains as a reconciliation marker.

Startup safety

During initialization, Pulse creates missing collections and named indexes, then inspects their keys and relevant options. Startup fails before workers begin when:

MongoDB permissions do not allow the required setup.
Existing data violates a required unique index.
A named Pulse index has incompatible keys, uniqueness, TTL, sparse, hidden, partial-filter, or collation configuration.

Pulse never drops or replaces an incompatible index automatically. The error identifies the collection, index name, expected configuration, and actual configuration so an operator can make the migration explicitly. When existing duplicate data prevents a unique index from being created, the startup error includes the affected collection, every unresolved Pulse index name, and MongoDB’s duplicate-key detail. See the API reference for the exact index names and configuration defaults.

Get Started

Essentials

Controllers

Other Modules

Reliability and recovery

Delivery lifecycle

Leases, heartbeats, and fencing

Recovery after a crash

Concurrent publication ordering

Crash windows

Design idempotent handlers

Error codes

MongoDB connection capacity

Polling

Collections and retention

Startup safety

​Delivery lifecycle

​Leases, heartbeats, and fencing

​Recovery after a crash

​Concurrent publication ordering

​Crash windows

​Design idempotent handlers

​Error codes

​MongoDB connection capacity

​Polling

​Collections and retention

​Startup safety

Delivery lifecycle

Leases, heartbeats, and fencing

Recovery after a crash

Concurrent publication ordering

Crash windows

Design idempotent handlers

Error codes

MongoDB connection capacity

Polling

Collections and retention

Startup safety