Sharding the OMS by User: Hot-User Actors with Eviction

The actor pattern for sharding shards a matching engine by order book (symbol) — the only consistency unit that matching can be partitioned on. This page is about the other layer: the per-user account / risk / session state that sits in front of the symbol-sharded matching engine. That layer can be sharded by user, and because a user’s state goes cold when they stop trading, it’s a natural fit for evictable actors — hydrate a user on their first order, dehydrate them when they go quiet.

The whole design is one idea applied relentlessly: the user-actor is a write-back cache over a durable DB, living inside an Aeron Cluster that must remain a deterministic state machine. Every decision below falls out of holding those two facts together.

Two different shard axes

Match by symbol, account by user. An order book is inherently multi-user and single-writer, so it can never be sharded by user. Account/risk/session state is per-user and can. This page shards the user layer; matching still happens in the symbol-sharded engine, which the user-shard forwards to.

What a user-actor owns — and why it’s a cache

A user-actor owns the reconstructable, per-user state needed to admit an order: session and sequence numbers, rate-limit counters, the open-orders index (which of this user’s orders are resting, and on which symbol/shard), and risk/buying-power as cached from the DB.

The database is the system of record; the actor is a hot cache. On a user’s first order the actor is hydrated (state read from the DB); when the user goes cold it is dehydrated (state flushed back to the DB, then dropped from memory). This is the Akka-cluster-sharding “passivation + persistence” shape, with one hard constraint that reshapes everything: it runs on a deterministic replicated state machine.

“Forwards to the ME” is a distributed transaction

The user-shard does not call the matching engine — it debits funds locally, acks the client, and emits the order as egress to a leader-only ME-bridge agent that is the actual ME client. The fill comes back asynchronously and settles. That saga — and the contract it forces on the ME — is the hardest part of the system; see Crossing to the matching engine.

The non-negotiable: I/O lives outside consensus

Aeron Cluster’s value is that the service is a deterministic state machine — every node replays the same log and must reach the same state, with no DB calls, no wall-clock, no RNG, no network inside the handler. A SELECT … FROM customer inside the clustered service would:

• make every follower stampede the DB (they all replay the same message), and
• on crash-replay, re-issue a read against a row that has since changed → nodes diverge, and
• block the single duty-cycle thread behind one user’s I/O, stalling every other user on the shard.

So all DB I/O happens outside the state machine, and results re-enter as logged messages:

• Hydration: the gateway / persistence tier reads the DB and injects a HydrateUser{userId, state, epoch} message onto the log, ahead of the order. The cluster only ever sees already-materialized, deterministic state.
• Flush: the cluster emits BeginDehydrate{userId, state, epoch} as egress; an external writer persists it (version-checked) and sends FlushComplete{userId, epoch} back as ingress.

The shard requests I/O; the tier performs it. The cluster never touches the DB. See deterministic state machines & Raft for the foundation.

Routing: two tiers

Every gateway must send all of a user’s orders to the one cluster that holds (or will hold) that user — the single-writer invariant. Two tiers:

• Whales — pinned. A curated whaleId → cluster map carried by every gateway, deliberately spread so two heavy users don’t collide. Reshard = move one entry (see tier changes).
• Normal users — rendezvous (HRW) hash. HRW(userId, membershipList) over the epoch-versioned cluster-membership list. No per-user directory: every gateway computes the same answer.

Why HRW, not hash % N. Plain modulo remaps ~(N-1)/N of all users the instant you add or remove a cluster — a mass split-brain, since each remapped user is still resident in its old cluster while new orders hash to a new one. HRW bounds churn to ~1/N and makes the moved set deterministically enumerable, so a scale event can migrate exactly those users.

The cluster is the routing authority. The gateway’s route is a correctable hint: a cluster NAKs orders for users it doesn’t own, and the NAK tells the gateway to re-route. Hydration and flush carry a monotonic fencing epoch per user, and the DB flush is a version-checked CAS (WHERE updated_at = expected) — so a partitioned gateway acting on a stale route, or a slow flush racing a newer write, cannot corrupt state.

Deciding a user is “cold” — deterministically

You asked for a keep-alive TTL: idle past T with no new orders → dehydrate. But wall-clock reads and free-running local timers are forbidden in the state machine — followers would read different clocks than the leader, and crash-replay would compare today’s clock to last-week’s log. The only legal clock is the cluster’s replicated (consensus) time, delivered as a logged timer message at a deterministic log position.

The low-latency structure is a hybrid LRU + TTL:

• Per order: O(1) — splice the actor to the MRU end of an intrusive recency list and stamp lastActiveClusterTime. (No per-user timer to reschedule — that would tax the hot path on every order, worst for your busiest users.)
• Periodic sweep (on consensus time): walk the cold tail only, stopping at the first non-expired actor — so the sweep is O(expired), not O(resident), and never causes an O(N) jitter spike on the duty-cycle thread. Cap evictions per sweep so a mass-expiry can’t stall matching.
• Size-bound safety valve: if resident count hits the memory cap, evict the LRU tail immediately, before TTL — protects against a cold-user flood OOMing the shard.

Cold ≠ no orders. “Flat” is three conditions.

“No new orders” is not “no live orders.” A user can rest a limit order then go silent; the order can fill at any microsecond. And once the credit-reservation saga exists, the actor also holds in-flight sagas (orders sent to the ME with no result yet) and outstanding reservations (debited funds). So eviction requires all three:

idle past TTL AND no resting orders AND no in-flight saga AND no outstanding reservation.

All are O(1) counter checks the actor already maintains. This keeps exactly the users who can generate fills resident, so the fill path is never cold (no DB read on a fill). A user with capital committed to the market simply isn’t cold — which is the correct behavior. A fill that races the eviction decision is caught by a backstop: a fill for a non-resident user triggers hydrate-on-miss, so it is never dropped.

The dehydration handshake

BeginDehydrate only marks the actor — its state is still in RAM. The flush is done off-thread by a colocated persistence agent; U is dropped from the resident map only when FlushComplete is consumed from the log (a deterministic position on every node).

• Order arrives during the flush window → cancel-on-activity. Because U’s state is still in memory, abort the eviction and serve from the live actor with zero penalty — no DB read, no parking. The actor bumps its epoch; a stale in-flight FlushComplete is then dropped as out-of-date. Lowest-latency outcome for the in-window order.
• Crash mid-flush is safe. U is removed only on FlushComplete, so a crash just leaves U resident on recovery and it gets re-evicted later. No lost write.
• Persistence agent is colocated, not gateway-side. Local Aeron IPC beats a gateway round-trip, and it isolates DB-write load off the order-forwarding path.

Hydration latency

A cold user’s first order can’t reach a resident actor — but you don’t want it to block the shard:

• The gateway hydrate-aheads: it already routed the order and knows U, so it fires the DB read and injects HydrateUser in front of the order — no shard→tier→shard round-trip.
• Single-flight / negative cache collapses a burst of a user’s first orders into one DB read.
• The order is parked and applied at a deterministic log position once HydrateUser lands — one async DB read for the cold order, while every other order on the shard keeps flowing.

Recovery

On restart a shard recovers by replaying its log + last snapshot, and the snapshot includes the resident actors. The resident set is a deterministic function of the log (HydrateUser/FlushComplete are logged), so recovery rebuilds the exact committed resident set with no DB reads — no post-restart stampede — and committed-but-unflushed (“dirty”) cache state survives in the snapshot rather than being lost. The DB is consulted only on a genuine cache miss for a cold user.

The snapshot must also contain in-flight sagas (orders sent to the ME with no result yet), and this is different from the eviction-flush. The eviction-flush (above) only ever serializes the quiescent subset of state, because eviction only happens when the user has no in-flight saga (the three-condition “flat”). The crash-snapshot serializes the full actor including sagas, because a crash can occur mid-saga — and the ME-bridge rebuilds its work queue from those resident in-flight sagas on failover. Snapshot-content ≠ flush-content.

Crossing to the matching engine: the credit-reservation saga

Everything above treats the user-shard as if it lived alone. It doesn’t: an order is born in the user-shard (where buying power lives) but matches in a different cluster — the symbol-sharded matching engine. “Forwarding the order to the ME” is therefore a distributed transaction across two independent Raft clusters, and it is the real hot path. The whole-system latency and most of the correctness live here, not in the LRU splice.

The OMS is the ME’s client — but the clustered service can’t be

The same rule that bans DB I/O in the handler (above) bans the ME client there too. The OMS handler runs on all 3–5 nodes; if the handler called meClient.offer(order) directly, the ME would receive the order 3–5 times, crash-replay would re-submit historical orders, and offer() backpressure would block the duty-cycle thread. An if (isLeader) offer() branch is also wrong — it puts a node-identity branch inside the deterministic state machine and still re-submits on replay.

So the order leaves through a leader-only ME-bridge agent — the egress twin of the persistence agent. The handler emits a deterministic SubmitToME egress message (identical on every node); Aeron delivers egress only from the leader, so the bridge running against the leader is the sole ME client — leader-only submission with no isLeader branch in business logic. The ME’s reply re-enters as a MEResult ingress message. (Routing is trivial: the order names its coin-pair, and symbol → ME is static config.)

Debit-then-send, with an immediate provisional ack

To keep the ME round-trip off the client’s critical path, the user-shard debits locally and acks immediately, then settles asynchronously:

• The debit is always exact. Limit buy = price × qty + fee; limit sell = base qty; market buy = the spend amount (quote-denominated, as Binance quoteOrderQty / OKX tgtCcy=quote_ccy) — you lock the spend, not the unknown receive; market sell = base qty. No worst-case band, no refund-remainder path on the buy side.
• “Accepted” is provisional. The ME may still reject (post-only would cross, price-band/lot-size, self-trade prevention, halted book), producing an async reject + credit-back. The UI must model pending → open | rejected. This is the deliberate price of moving the ME chain off the client’s critical path — a product/UX decision, not just an architecture one.
• Reserve / settle / release are four logged, idempotent transitions. available = total − Σ reservations; settle filled legs at the real fill price; release on cancel/reject/expiry and on the unfilled remainder of a partial fill. A dropped settle/release event is a money bug — guarded by reconciliation (below).

The ME is a protocol participant, not an opaque target

This is the load-bearing dependency: exactly-once order handling across two independently-failing clusters is impossible without a contract on the ME. “The OMS is just a client and the ME is untouched” is false. The ME must:

1. Accept a caller-supplied deterministic orderId (e.g. omsClusterId:userId:userSeqNo — derivable on replay, never a random UUID) and deduplicate on it for the order’s lifetime.
2. Make accept/fill state idempotently re-queryable by that id — re-submitting an already-accepted orderId returns the existing order, not a second one.
3. Emit exactly-once, ordered lifecycle events for every transition — including the ME’s own autonomous expiries/cancels (GTD/GTT expiry, kill-switch, halts) — and expose a per-user open-orders snapshot for reconciliation.

Why each matters: (1)+(2) let the bridge re-submit the entire in-flight set on OMS-leader failover without risking duplicate fills (dedup-key + safe-retry — the textbook exactly-once construction). (3) keeps the OMS open-orders index — a replica of ME state — true: a dropped ME-side expiry would otherwise leave a phantom resting order, so the user is never “flat”, never evicted (defeating the whole cold-eviction premise), and the reservation behind it leaks forever (frozen funds).

Highest-priority external dependency

If the ME team cannot deliver requirements (1)–(3), this design cannot be built correctly — it degrades to at-most-once order submission (silently dropped orders), which an exchange cannot ship. Treat the ME contract as a cross-team commitment, not an OMS-internal detail.

Backpressure, failover, and the bridge’s durable work queue

The bridge’s offer() is non-blocking and can fail (back-pressured, ME not connected). The bridge is not the source of truth — the resident set of in-flight (PENDING_RESERVE) orders is, and it is reconstructable from log + snapshot. So:

• PENDING_RESERVE is the bridge’s durable work queue. Its in-memory retry queue is just a cache; its real job is “drive every in-flight order to the ME until its MEResult returns.” On bridge crash or OMS-leader failover, rebuild from state and idempotently re-drive every pending order — the ME’s dedup (contract #1) makes unlimited retries safe. A PENDING_RESERVE after failover is indistinguishable between “never reached the ME” and “already there”; re-submitting and letting the ME dedup resolves it.
• Upstream backpressure. When the in-flight count crosses a threshold (ME slow or down), the OMS stops accepting new orders (NAK at ingress / tell the gateway) rather than debiting users for orders it can’t deliver — which would lock capital behind an unreachable ME.

Reconciliation: the money-leak backstop

Event streams will drop a message over months, and a stuck reservation never self-heals. A periodic, capped, deterministic-input reconciliation sweep (the sibling of the eviction sweep) compares the ME’s per-user open-orders snapshot (contract #3) against the OMS index and reservations, and heals drift: close phantom orders, release leaked reservations, flag impossible states.

Tier changes & scaling

Because state lives in the DB, no movement ships state between clusters. Every move — tier change or scale event — is the same fenced primitive:

quiesce U → flush to DB → evict from source → atomically flip routing (one epoch) → U cold-hydrates on destination from the DB on its next order.

• Tier change (normal ↔ whale) is an admin operation, never auto/volume-based. On admin commit, run the fenced migration for that one user in real time; coalesce rapid successive admin changes into a single epoch bump. It’s a rare single-user op — batching would only concentrate impact into a spike.
• Scale in/out changes HRW membership for ~1/N of normal users at once — inherently a batch, so the lever is to de-concentrate it. The new cluster joins as receiving (HRW won’t route new cold users to it until active, controlling fill rate); a controller migrates affected users in capped-concurrency waves with per-wave fencing; idle affected users migrate lazily (their next order re-routes under the new epoch and cold-hydrates on the new owner) so only resident affected users are actively drained. This trades longer wall-clock (unfelt) for a flat hot-path profile (no cohort-wide p99 spike).
• One atomically-versioned routing config + epoch. The membership list and whale overrides are distributed together, so a gateway is always fully on epoch N or N+1 — never half, which would double-resident a user.

See scaling the Aeron Cluster and rolling upgrades for the per-cluster mechanics each wave relies on.

Invariants

1. Single-writer per user — resident in at most one cluster at any instant. Everything else exists to protect this.
2. No external I/O inside the state machine — DB, clock, RNG, network all live outside consensus; results re-enter only as logged messages.
3. DB is the system of record; the actor is a write-back cache — money is flushed (version-checked CAS) before U is dropped, and U is dropped only on FlushComplete.
4. The cluster is the routing authority — gateways are correctable caches; a cluster NAKs orders for users it doesn’t own.
5. One atomically-versioned routing config — membership + whale overrides move together under one epoch.
6. Determinism per shard — eviction uses the consensus clock; hydration and eviction are logged, ordered events applied at deterministic log positions.
7. Exactly-once order submission across two clusters — a deterministic orderId + ME-side dedup; the bridge may re-submit any in-flight order freely and the ME absorbs duplicates.
8. The OMS is the ME’s client, never inside the ME — the handler emits orders as egress; a leader-only ME-bridge agent is the sole ME client. No offer() and no isLeader branch in the handler.
9. Debit-first, debit-exact, release-the-remainder — funds are debited locally before the order goes to the ME (exact, because market buys are quote-denominated); fills settle at the real price and unused frozen funds are released. A dropped release/settle is a money bug — guarded by reconciliation.
10. “Flat” = no resting orders AND no in-flight saga AND no outstanding reservation — eviction requires all three.

Anti-patterns

• DB I/O inside the clustered handler — follower stampede, divergent replay, a blocked duty cycle.
• hash % N over physical clusters — every topology change is a mass migration.
• Wall-clock TTL or local timers in the handler — nondeterministic; followers and replay diverge.
• Evicting a user with live resting orders — a fill then lands on a missing actor (or pulls liquidity if you “fix” it by cancelling).
• Two separately-versioned routing maps — a gateway sees a half-applied state and double-residents a user.
• Per-order least-busy routing — re-places a resident user mid-session → double-hydrate → double-spend. Load can pick initial placement; it must then be sticky.
• Calling the ME client from the handler (or if (isLeader) offer()) — duplicate submits across nodes, re-submits on replay, blocks the duty cycle. Emit egress; let a leader-only bridge submit.
• Treating the ME as untouchable — without caller-keyed dedup + lifecycle events + a reconcile snapshot, you can’t get exactly-once across the two clusters, and reservations leak on dropped events.
• Evicting a user with funds committed to the market — an in-flight saga or outstanding reservation means the user is not flat; evicting drops the actor a MEResult is about to hit.
• Final-acking an order before the ME confirms — the ME can still reject; “accepted” must be provisional, with an async reject + credit-back path.

---

Related: actor-pattern (symbol) sharding, cluster & Raft overview, scaling the Aeron Cluster, cluster standby & HA design.