Architectural R&D specimen 02 — Reference: enterprise integration layer
Fail-safe event queue: durable acceptance, bounded retries, idempotent effect
Architectural R&D specimen: studio research and development, built and measured in this repository. It is not client work and describes no client. Every figure on this page is measured or computed, and its method is stated.
Design question
When one system in a chain stalls, events must neither vanish nor be applied twice. Can a small, dependency-free state machine give at-least-once delivery with an at-most-once effect, survive a restart and a dead channel, and show it on deterministic failure scenarios?
Evidence tier B: a reference implementation verified by executable tests. It is not wired to any client’s systems; volumes, credentials and legal context are outside what this specimen claims.
I
Constraint and tolerance
Three failure modes, each given a written tolerance.
Loss: an event acknowledged to its producer must survive a restart and a dead channel. Acceptance is a durable write that happens before the acknowledgement.
Repetition: networks repeat. Delivery is at least once, so the receiver applies an event id once and drops the repeat, whoever sent it again.
Unboundedness: no event may be retried without end or held without end. Attempts, age and the outcome at each limit (dead-lettered, kept, never dropped) are written down as policy.
II
Architecture
A state machine with no timers and no input or output of its own.
The caller supplies the clock, the durable store and the channel, so the same code runs in a deterministic simulation, in the browser and, with a real store and channel, in a project.
States: pending, retry, parked, delivered, dead. Failed sends retry with exponential backoff. After a run of failures and a failed health probe the circuit opens: events are parked in the store and a cheap probe tests the channel until it returns, then the backlog drains in order.
Poison isolation: a failing event on a healthy channel is a bad event, not a dead channel. It is retried within its budget, then dead-lettered, and the circuit stays closed, so one bad event does not stop the queue behind it for longer than that budget.
III
Verification
Deterministic scenarios and a fixed set of invariants.
Scenarios include a channel down for twenty seconds, a flapping channel, acknowledgements lost on 30 % of sends, a poison event, a stubborn event, restarts during an outage, a producer that sends twice, and everything at once. Each uses a virtual clock and a seeded generator: no timers, no network.
On every scenario the contract tests assert that no accepted event is lost, none is applied twice, the order of acceptance is kept and no event is sent more often than the policy allows.
The results table in the documentation is generated from the code and compared with it by a test, so the evidence is what the code does today.
Where the tiers apply: this queue is evidence tier B (a reference verified by tests). The signed webhook with idempotent delivery that this site runs in production is tier A.
IV
Measured result and limits
Computed from the code that the tests run.
Measured
These figures are computed when this page is drawn, by running the scenarios the contract tests run. They are deterministic: the same on every request.
- Scenarios run
- 9
- 9 of 9 held every invariant
- Events accepted
- 370
- across all scenarios
- Events lost
- 0
- every accepted event applied, dead-lettered or still queued
- Events applied twice
- 0
- 56 repeats were dropped
Computed from lib/arch/scenarios.ts by the simulator the contract tests run, when this page is drawn. 5 events were dead-lettered (kept, never dropped): poison or stubborn events, by design.
Trade-offs and limits
What this pattern costs, and where it needs the client.
At least once means repeats
Repeats happen by design, so the receiver must cope with them and must remember ids for longer than the longest time the queue can keep retrying.
Order costs availability
With strict order, one failing event delays the ones behind it until it succeeds or is dead-lettered. Switch it off for events that are independent.
The store is a dependency
Durability is only as strong as the store the queue is given. Its choice (a database, a managed queue) and its own guarantees belong to the project.
Measured on
- TypeScript · no dependencies
- Virtual clock · seeded generator
- Node · browser