> ## Documentation Index
> Fetch the complete documentation index at: https://www.orionjs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Reliability and recovery

> Delivery guarantees, leases, fencing, reconciliation, retention, and idempotency

Pulse uses MongoDB documents, unique indexes, renewable leases, and reconciliation to provide recoverable delivery without requiring multi-document transactions.

## Delivery lifecycle

For every event and consumer group, Pulse follows this lifecycle:

```text theme={null}
event
  → durable delivery (pending)
  → history attempt (pending, no lock)
  → history attempt (pending, leased)
  → handler
  → history attempt (success or error)
  → delivery (success, retry, or terminal error)
```

The history attempt exists before the handler starts. Operators can therefore distinguish work that is queued, currently running, or abandoned by a dead worker.

| History state                           | Meaning                                                          |
| --------------------------------------- | ---------------------------------------------------------------- |
| `pending` without `lockedUntil`         | Queued or waiting for `nextAttemptAt`.                           |
| `pending` with a future `lockedUntil`   | A worker is executing it.                                        |
| `pending` with an expired `lockedUntil` | The worker or machine stopped renewing its lease.                |
| `success`                               | The handler completed and Pulse acknowledged the attempt.        |
| `error`                                 | The handler failed, the worker was lost, or the payload expired. |

## Leases, heartbeats, and fencing

When a worker acquires an attempt, it writes:

* `startedAt`, `lockedAt`, and `lockedUntil`
* a UUIDv7 `lockOwner`
* a new UUIDv7 `lockToken`
* `heartbeatAt`

Pulse renews the attempt lease approximately every third of `lockTimeoutMs`. Ordered consumers also renew the topic ordering lease.

Every success or error update includes the fencing token in its MongoDB filter. If another replica has already recovered the attempt, the stale worker's token no longer matches and it cannot overwrite the new result.

Choose a `lockTimeoutMs` that comfortably exceeds expected event-loop stalls and temporary MongoDB connectivity interruptions. Heartbeats keep normal long-running asynchronous handlers alive, but blocking the JavaScript event loop can prevent renewal.

## Recovery after a crash

Every worker loop runs three recovery steps before looking for new events:

1. Reap one expired attempt.
2. Reconcile incomplete delivery state.
3. Acquire existing work or discover new events.

An expired attempt is atomically changed from `pending` to `error` with code `worker_lost`. At-least-once mode then creates the next numbered attempt; at-most-once mode makes the delivery terminal.

Reconciliation repairs safe partial states idempotently:

* Delivery exists but its first history attempt does not.
* History is successful but delivery is still pending.
* History is errored but its retry was not created.
* Event discovery was interrupted before the subscription cursor advanced.

Unique indexes on `consumerGroup + eventId` and `deliveryId + attempt` ensure replicas converge on one logical record.

## Crash windows

| Crash point                                       | Recovery                                                      | Handler may run again?                       |
| ------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------- |
| Before the handler starts                         | Expired lease becomes `worker_lost`; retry is created.        | Yes, but the previous handler did not start. |
| During the handler                                | Expired lease becomes `worker_lost`; retry is created.        | Yes.                                         |
| After a side effect but before history success    | Another attempt is created.                                   | **Yes.**                                     |
| After history success but before delivery success | Reconciliation finishes the delivery from successful history. | No.                                          |
| After history error but before retry creation     | Reconciliation creates exactly the next numbered attempt.     | Yes.                                         |

At-least-once delivery cannot prove whether an external side effect happened before a machine disappeared. Exactly-once side effects require idempotency in the system receiving the side effect.

## Design idempotent handlers

Use `event.id` as the idempotency key sent to an external API:

```typescript theme={null}
await pulse.subscribe('payment.captured', async event => {
  await ledger.recordPayment({
    idempotencyKey: event.id,
    paymentId: event.data.paymentId,
    orderId: event.data.orderId,
  })
})
```

When the side effect is in MongoDB, store the event ID with a unique index in the same transaction as the domain write:

```typescript theme={null}
await mongoClient.withSession(async session => {
  await session.withTransaction(async () => {
    await processedEvents.insertOne(
      {_id: event.id, processedAt: new Date()},
      {session},
    )
    await orders.updateOne(
      {_id: event.data.orderId},
      {$set: {invoiced: true}},
      {session},
    )
  })
})
```

Treat a duplicate-key error on `processedEvents._id` as an already completed handler. Do not acknowledge the idempotency marker before a non-transactional side effect that can still fail.

## Error codes

Pulse writes these built-in error codes to history:

| Code            | Meaning                                               |
| --------------- | ----------------------------------------------------- |
| `handler_error` | The handler threw or rejected.                        |
| `worker_lost`   | The attempt lease expired before completion.          |
| `event_expired` | Retention removed the event payload before execution. |

Internal errors and lost-fencing notifications are delivered to `onError`. The callback should report or log quickly and must not throw:

```typescript theme={null}
const pulse = connect<AppEvents>({
  connectionString,
  databaseName: 'commerce',
  consumerGroup: 'billing',
  onError(error) {
    logger.error('Pulse worker error', {error})
  },
})
```

## Polling and Change Streams

Polling and reconciliation guarantee progress. Change Streams only wake sleeping workers after new event inserts.

| `changeStreams` | Behavior                                                                                         |
| --------------- | ------------------------------------------------------------------------------------------------ |
| `auto`          | Enable on replica sets or sharded clusters; fall back to polling on standalone MongoDB. Default. |
| `required`      | Fail `awaitConnection()` unless MongoDB supports Change Streams.                                 |
| `disabled`      | Use polling only.                                                                                |

If an active Change Stream fails, Pulse reports the error and attempts to open it again after `pollIntervalMs`. Missing a notification does not lose the event.

## Collections and retention

With the default prefix, Pulse owns:

| Collection                    | Retention                                       |
| ----------------------------- | ----------------------------------------------- |
| `orionjs.pulse.events`        | `eventRetentionMs` from publication time.       |
| `orionjs.pulse.subscriptions` | Durable; no TTL.                                |
| `orionjs.pulse.deliveries`    | `historyRetentionMs` after terminal completion. |
| `orionjs.pulse.history`       | `historyRetentionMs` after the attempt ends.    |

Retention uses absolute `expiresAt` dates and TTL indexes with `expireAfterSeconds: 0`. When retention is `null`, Pulse omits `expiresAt`; MongoDB keeps those documents even though the TTL index remains.

## Startup safety

During initialization, Pulse creates missing collections and named indexes, then inspects their keys and relevant options. Startup fails before workers begin when:

* MongoDB permissions do not allow the required setup.
* Existing data violates a required unique index.
* A named Pulse index has incompatible keys, uniqueness, or TTL configuration.
* `changeStreams: 'required'` is used on an unsupported topology.

Pulse never drops or replaces an incompatible index automatically. The error identifies the collection, index name, expected configuration, and actual configuration so an operator can make the migration explicitly.

See the [API reference](/overview/other-modules/pulse/api-reference) for the exact index names and configuration defaults.