Sharding an OMS by the Actor Pattern

A clustered matching engine is a single deterministic state machine: it consumes the replicated log one message at a time, on one logical thread, in one fixed order. That is exactly what makes it replayable and crash-safe — and it is also the ceiling. One cluster’s order-matching throughput is bounded by what a single core can process sequentially. When one machine is no longer enough, you don’t make the state machine concurrent (that breaks determinism); you run more of them, each owning a slice of the instrument universe. The actor pattern is the discipline for doing that cleanly.

The actor as a shard

An actor is a single-threaded unit that owns a partition of state and processes its inbound mailbox one message at a time, with no shared mutable state and no locks. That description is already the Aeron Cluster service: the message handler is the actor, and the replicated log is the mailbox. Sharding by the actor pattern just makes this explicit and replicates it N times — each actor owns a disjoint set of order books and never reaches into another actor’s state.

The properties that make it work:

• Single-writer per book. All orders for a given symbol are totally ordered by exactly one actor. There is never contention for a book, so there are no locks to tune and no race conditions to reason about.
• Determinism is preserved per shard. Each actor remains a deterministic state machine (see below), so each shard is independently replayable.
• Parallelism comes from count, not concurrency. Throughput scales by adding shards, not by adding threads inside a shard.

Two ways to realize it on Aeron

Option A — one Aeron Cluster per shard. Each shard is an independent Raft group with its own replicated log, its own leader, and its own snapshot. This is the only option that actually scales throughput horizontally: each shard’s log is consumed on a different host’s core, so total matching capacity grows with shard count. It also gives independent blast radius (one shard’s failover doesn’t touch the others) at the cost of more operational surface — more leaders to place, more logs to monitor, more upgrades to roll.

Option B — many actors multiplexed inside one cluster. A single clustered service routes each inbound message to an in-process actor keyed by shard. This is clean for code isolation and state organization, but it does not scale throughput: the single Raft log is still consumed sequentially on one core, so all actors share that one core’s budget. Use it to keep books logically separated within a capacity envelope you already know fits one cluster — not to go faster.

Rule of thumb: isolation of logic → Option B is fine. Isolation of load → you need Option A. Mixing them (a handful of clusters, each hosting several logically-separated actors) is common and reasonable.

Multiplexing multiple books in one cluster (Option B, in detail)

When a cluster hosts several symbols, the clustered service doesn’t need a fundamentally different design — it multiplexes by symbol. Because order books are naturally independent (a match in BTC/USDT never touches the ETH/USDT book), this is almost pure partitioning — the easy direction, in sharp contrast to the cross-shard account concerns below.

What changes versus a single-book cluster:

• Dispatch by symbol. The handler reads the order’s symbol and routes it to that symbol’s in-process OrderBook — a symbol → OrderBook map. One handler, N books.
• Per-symbol state, cleanly partitioned. Each book is its own structure (bids/asks/sequence). No book reads or writes another’s state — the same single-writer discipline, now N single-writers living on one thread.
• Snapshot = every book in the cluster. The Aeron snapshot serializes all resident books, so snapshot size and recovery time grow with symbols-per-cluster — a real capacity-planning cost.
• Per-symbol timers on the shared consensus clock. GTD/GTT expiry, auction schedules — each book arms its own timers, but they fire on the one cluster’s replicated time and are processed on the one thread.

What does not change — and is the whole point of Option B — is that all books share one Raft log, one leader, and one core, consumed sequentially. Adding books organizes state; it does not add throughput. Two consequences:

• Fairness on the shared thread. A burst on one hot book can starve the latency of the others, since they queue behind it in log order. For most venues, processing in committed-log order is fine; if it isn’t, you need a fair-dispatch/batching policy across books — or you’ve outgrown Option B for that symbol and should isolate it (Option A).
• Placement is the real decision. Co-locate low-volume or correlated pairs (they fit one core’s budget, may share auction logic); isolate a hot symbol onto its own cluster. That is exactly the symbol → shard map in Choosing the shard key.

Choosing the shard key

The shard key must be the unit of consistency, and for a matching engine that unit is the order book — every order for one symbol must be totally ordered against every other order for that symbol. So you shard by instrument / symbol, never below it.

• Static partition map (symbol → shard) is the default. It is auditable, lets you deliberately co-locate correlated instruments, and lets you isolate a hot symbol onto its own shard. The cost is that you own the balancing decisions.
• Consistent hashing removes manual placement but takes away that control — and a hash can’t honor “these two symbols trade together, keep them on one shard.” For a venue with a known, slowly-changing instrument list, a versioned static map usually wins.
• A hot symbol cannot be split. Its book is intrinsically single-threaded. The lever you have is placement — give a very active symbol its own shard rather than trying to parallelize its book.

Routing: the gateway and the map

The ingress gateway maps each order to its owning shard and forwards it to that shard’s cluster ingress. The routing map is load-bearing shared truth: gateways and shards must agree on the same versioned map. Drift — a gateway sending symbol X to shard 2 while shard 1 still believes it owns X — is the classic split-brain-by-configuration bug. Version the map, distribute it atomically, and have shards reject orders for symbols they don’t own rather than silently accepting them.

Cross-shard concerns (the hard part)

Per-symbol sharding is clean until something spans symbols. The three usual offenders:

• Account balance / margin / risk. Buying power is per account, not per symbol, so it doesn’t live naturally in a per-symbol shard. The common shape is a separate risk/account actor sharded by account, with a pre-trade credit reservation: the order shard reserves against the account shard before matching, and releases or settles after. This is a saga, not a distributed transaction.
• Multi-leg / basket orders that touch several books need a coordinator that drives the legs and compensates on partial failure — again a saga, with all of its steps flowing through the logs so the whole thing stays replayable.
• Self-trade prevention across an account whose orders land on different shards needs the account-level actor (or a shared, replicated STP check) rather than a per-book one.

The guiding principle: keep the hot path single-shard. Cross-shard coordination is for the exceptions; if your common order needs two shards to agree synchronously, the shard boundary is in the wrong place.

Determinism does not relax — it multiplies

Sharding does not buy you any slack on the rule that makes Aeron replay work. Every actor must still be a deterministic state machine: no wall-clock reads, no RNG, no external I/O, no nondeterministic iteration order inside the handler. (See deterministic state machines for the foundation.) Two sharding-specific corollaries:

1. Cross-shard coordination must flow through the logs. A reservation message from an order shard to an account shard has to be a real, logged ingress/egress message on both sides — never an out-of-band RPC — or you’ve introduced a replay-breaking side channel.
2. Time and sequencing are per shard. Each shard has its own log position and its own notion of “now” (derived deterministically from its log). Don’t assume two shards’ timelines are comparable; if you need a global ordering across shards, you must build it explicitly.

Resharding: moving a symbol between shards

Rebalancing is a mini-migration, and it is the riskiest routine operation in a sharded OMS. The safe shape:

1. Quiesce the symbol on its source shard (stop accepting new orders for it; let in-flight work drain).
2. Snapshot / serialize that book’s state.
3. Hand off the serialized state to the target shard and let it load.
4. Flip the routing map atomically — bump the version so gateways start sending the symbol to the new owner, and the old owner starts rejecting it, at the same logical instant.
5. Resume on the target.

Capacity-plan so you do this rarely: pick shard count and symbols-per-shard with headroom, and keep hot symbols isolated so a single book’s growth never forces a reshard.

Failover & operations

• Per-shard Raft. Under Option A each shard elects its own leader. Spread leaders across hosts and AZs so losing one host doesn’t simultaneously decapitate many shards — see cluster standby & HA design.
• Monitor per shard. Commit position, backpressure on each shard’s ingress, NAK/error counters — one quiet aggregate can hide one sick shard.
• Upgrades multiply by shard count. A rolling upgrade is now N rolling upgrades; automate it and apply the same follower-first, leader-last discipline from rolling upgrades to each shard.

Anti-patterns

• Shared mutable state between actors. If two actors can touch the same field, you no longer have actors — you have a lock you forgot to write.
• Synchronous cross-shard calls on the hot path. Turns independent shards into a distributed lock-step system with the latency of the slowest participant.
• Routing-map drift. Gateways and shards disagreeing about who owns a symbol. Version the map and make ownership authoritative on the shard.
• Splitting a single hot book. It can’t be done while preserving total order — isolate it instead.

---

Related foundations on this site: cluster & Raft overview, scaling the Aeron Cluster, performance tuning, and the section’s rolling upgrade procedure.