Aeron® Cluster Snapshot Performance: A Matching-Engine Case Study

A matching engine hosted as an Aeron® Cluster ClusteredService is single-threaded: the same service thread that applies every sequenced command also runs onTakeSnapshot. While the cluster snapshots, the book cannot trade, and every order that arrives meanwhile queues behind it. That is a tail-latency (p99/p999) event you schedule yourself. Two numbers measure it, and they differ:

• the take, the time the service thread spends writing the snapshot (§1–9 drive it down);
• the client pause, how long a client actually waits (§10). The take is part of the pause, not all of it.

The engine in this case study started out unable to snapshot more than about 16 resting orders. On a real 3-node cluster at production durability (file sync level 2), the Archive’s per-block fsync, not encoding, set the pause clients saw, until the Archive block went from 1 MB to 16 MB.

Key numbers

result	value	conditions	§
Take, 2M orders, single-threaded	977 ms (first working version) → 332 ms	live single-node cluster, G1, Archive on tmpfs; identical stateHash after every restore	result
Linear take on ZGC vs G1	about 2.3× slower on ZGC	ZGC is the production collector	§8
Take, 5M orders, ZGC	1,490 ms linear → 179 ms parallel (p99 196 ms)	bench, 8 threads, dedicated c7i.4xlarge	§9
Client pause, 5M orders, 1 MB Archive block (Aeron default)	~2.4 s, linear or 8-thread encode alike	3 × c6in.16xlarge, file sync level 2, gp3	§10
Client pause, 5M orders, 16 MB Archive block	~0.8 s with 8-thread encode, ~1.29 s linear	same, gp3 at 1,000 MB/s	§10
Client pause, 10M orders, 16 MB Archive block	~1.35 s	same	§10
Heap per resting order	201–209 B on G1, 238–245 B on ZGC	TreeSet book, 1M–10M orders	§7

Note

Stack: Aeron 1.53.3, SBE 1.40.2, Agrona 2.6.1, Java 25. The page uses three environments:

• §1–7: an 8 vCPU / 30 GB box shared with other workloads, runs on 2026-10-14, all on G1 (the test JVM’s default).
• §8–9: a quiet, dedicated c7i.4xlarge with production’s ZGC flags (and G1 alongside in §8).
• §10–11: a 3-node c6in.16xlarge cluster with the production node configuration.

Bench figures show scaling shape; the §10 client-pause figures are the production claim. For how snapshots are recorded and replayed under the hood, see The Aeron Files. For what happens when a snapshot fails halfway, see Snapshot Failure and Recovery.

Take time and client pause are different numbers

Sections 1–9 report the take: the time the service thread spends writing the snapshot. The parallel and ZGC takes use a drain-only IPC reader, and the single-node “live” runs, including the recording-log and restart columns of the result table, wrote the Archive to tmpfs (RAM), not EBS. Section 10 reports the client pause: the worst latency a client saw around each snapshot on a real 3-node cluster with the Archive on gp3. At production durability that pause is set by the Archive’s fsync, so a faster take does not shorten it until the Archive is fixed. The production claim at file sync level 2 is the 16 MB-block, 1,000 MB/s row in §10.

The result

All rows are a 2M-resting-order book on a real single-node cluster on G1, with the Archive on tmpfs: snapshot → Archive → truncate the log behind the snapshot → restart → restore. For production’s ZGC and for parallel encode, see §8 and §9.

Stage	Take	Trigger → recording log¹	Restart → LEADER²
Original code	fails above ~16 resting orders	—	—
Chunked, DataOutputStream serializer	977 ms (759 serialize + 218 publish)	1,166 ms	1,906 ms
ByteBuffer serializer	417 ms (186–232 + 182–231)	~600 ms	~1.9 s
SBE, zero-copy, one pass	366 ms	536 ms	1.2 s
+ 64 KB snapshot-channel MTU	332 ms	495 ms	1.2 s

¹ From ClusterTool.snapshot until the SNAPSHOT entry appears in the recording log (polled every 20 ms). ² From node launch on the same directory, through Archive replay and restore, until LEADER.

Other measured figures for the reference engine (a TreeSet book):

• Heap: about 201–209 B per resting order on G1 and 238–245 B on ZGC (production), measured from 1M to 10M orders (§7). ZGC can’t use compressed oops, so it needs about 18% more.
• Wire size: about 82 B per order in the final format.
• Allocation: a take allocates 1 KB in total, whatever the book size.

1. One claim for the whole book caps you at 1,376 bytes

The original onTakeSnapshot serialized the whole book to a byte[], copied it into a second full-size buffer, and wrote it with a single tryClaim:

final byte[] payload = me.snapshot();                     // DataOutputStream over a ByteArrayOutputStream
buf = new UnsafeBuffer(new byte[12 + payload.length]);    // second full-size copy
publication.tryClaim(len, claim)                          // ONE claim for the whole book

ExclusivePublication.tryClaim throws if the length exceeds maxPayloadLength, which is MTU − 32. The cluster’s default service snapshot channel is aeron:ipc?alias=snapshot, and the IPC MTU defaults to 1,408 bytes, which gives a 1,376-byte cap. At 85 B per order the snapshot threw at about 16 resting orders. The only real-Aeron snapshot test used 8 orders, so nothing caught it.

Switching to one offer() only moves the ceiling to maxMessageLength (term length / 8). That is still far short of millions of orders, and restore would then need a fragment assembler.

Size one test past every framing limit

A snapshot test with a toy-sized state proves the plumbing, not the design. Size at least one test past every framing limit you can name: MTU, max message length, term length.

2. Chunk first, and find the O(n²) hiding elsewhere

The first fix was correctness: split the snapshot into frames that each fit one claim, each carrying {magic, totalLength, offset}, and reassemble them on restore. Three supporting changes went with it:

• Back-pressure: retry BACK_PRESSURED and ADMIN_ACTION without limit and throw on anything else. The old bounded retry silently dropped part of the snapshot.
• Client keepalive: the service container’s Aeron client runs in invoker mode. A multi-second snapshot loop that never drives the conductor’s AgentInvoker misses its driver keepalives and gets timed out. Invoke it while waiting and every 256 claims; Aeron’s own ServiceSnapshotTaker does the same.
• Fail loudly: an incomplete snapshot now fails the restart. Before, an unrecognised snapshot booted an empty book with no error.

Two more problems surfaced only once books were genuinely large:

Found by	Problem	Fix	Effect
Live cluster: 200k orders could not be built in 50 s	After every command the service published top-10 depth via displayedLadder(), which walked every resting order to build the whole ladder, then truncated it. Each command was O(book), so filling a book was O(n²).	Stop the walk after depth levels	>50 s (never finished) → 2.7 s
The bench	The array-backed engine’s ArrayBook.add scanned from the front, so rebuilding a book in order was O(n²)	Append when the order sorts after the tail	Restore at 200k: 45 s → linear

Tip

At millions of orders, anything O(book) per command is the outage, not the snapshot. A “top-N” read that builds all N and then truncates is a classic hidden O(n).

3. The serializer: two bad constant factors and one bad complexity

jstack sampling of the old snapshot() at 1M orders showed it was bound neither by I/O nor by the walk. Three things cost the time:

1. resting.sort(comparingLong(o -> o.orderId)) over every order: O(n log n), with a cache miss on every comparison.
2. bookInstrumentOf(o) for every order: a TreeSet.contains against every book, just to rediscover which instrument the order belongs to.
3. DataOutputStream over ByteArrayOutputStream: a synchronized write per field, doubling-and-copy growth, then toByteArray copying the whole result once more.

The fixes:

• Walk instead of sort. Emit book by book (instrument ascending), bids then asks, in the book’s own price-time order. The output is still deterministic, with no sort and no lookup. Restore re-inserts into the same comparator-ordered TreeSet, so snapshot → restore → snapshot stays byte-identical. At 1M orders (bench), serialize went from 6.5 s to 491 ms. The 6.5 s was measured on a contended box (load average 9–54 on 8 CPUs), so treat it as an upper bound; the 491 ms is from a quiet box. The cleanest quiet-box figure is live: at 2M orders (load average 0.3), serialize was 759 ms after this fix and before the ByteBuffer change below. That is the 977 ms row in the result table.
• Exact pre-sizing. Every field is fixed-width, so the buffer can be sized up front. Allocation at 1M orders dropped from 344 MB to 81 MB.
• Replace DataOutputStream with a heap ByteBuffer. It is big-endian too, so the bytes and the restore code are unchanged. A standalone split at 2M orders isolated the costs:

2M orders	Time
snapshot() with DataOutputStream	~850 ms
Walking the book only	~110 ms
Walk + ByteBuffer puts of the same 85 B	~170 ms

About 80% of serialize time was stream overhead. In the live cluster, serialize went from 759 ms to 186–232 ms.

Profile before redesigning

The serialize time was mostly self-inflicted constant factors (a sort, a reverse lookup, synchronized stream writes), not the cost of the data.

4. Aeron’s own pattern: Begin / batches / End, zero-copy, one pass

With a fast serializer the take was still two passes on the service thread: walk the book into a 170 MB byte[], then copy that into 125,000 claims. Aeron Cluster’s own ServiceSnapshotTaker streams state as many small typed messages framed by begin and end markers. The engine adopted the same shape:

• A sink interface with no Aeron or SBE dependency. SnapshotSink has one callback per record type, taking only primitives so nothing is boxed per order: onHeader, onSourceHighWater, onMark, onResting, onArmedTrigger, onEnd. Each matcher implements snapshotTo(sink) and restorer(), which resets the matcher and returns a sink that rebuilds it.
• SBE wire format. Six messages: MeSnapshotBegin, SourceHwm, Mark, Orders, Triggers, End. Orders and triggers are repeating groups sized to fill exactly one tryClaim. A batch is claimed at full size and encoded in place as the walk goes. When the record type changes or the snapshot ends, the batch is closed early with resetCountToIndex(). No message ever exceeds one claim, so nothing is fragmented and restore needs no assembler.
• The publisher is the sink. The matcher walks its book once, and each onResting writes the order’s 13 fields straight into the claimed term-buffer region through the SBE flyweight. There is no snapshot byte[] and no copy. Measured with ThreadMXBean.getThreadAllocatedBytes from 10k to 4M orders, the reference engine’s take allocates 1 KB at any book size.
• Strict restore. The loader decodes each message in place from the replayed image and feeds the restorer. It requires Begin first and End last, and End’s counts must match what arrived.
• One record API. The in-process snapshot() / restore(byte[]) became default methods on top of the same record stream, so the existing in-process tests and the cross-engine differential harness exercise the same record API the cluster uses.

Live at 2M orders: take 417 → 366 ms. Restart dropped from 1.9 s to 1.2 s, because restore no longer reassembles a 170 MB array before decoding.

Image.poll swallows your handler’s exceptions

Aeron 1.53’s Image.poll wraps the onFragment call in catch (Exception) → errorHandler.onError(...) and returns normally. A loader that rejects a malformed snapshot by throwing from onFragment never stops the restore loop, so a restarting node spins forever in onStart instead of failing. The new failure-path test caught it by hanging.

The fix: the handler records its first failure in a field and stops applying further fragments, and the polling loop rethrows after poll returns. Never rely on an exception escaping Image.poll or Subscription.poll; validation inside a fragment handler must hand its verdict back to the loop.

5. A 64 KB snapshot MTU: 50× fewer messages, about 10% faster

Because maxPayloadLength is MTU − 32, one Orders frame at the default 1,408-byte MTU holds 16 orders of 82 bytes. Raising the service snapshot channel to the IPC maximum gives about 800 orders per frame:

ClusteredServiceContainer.Context

context.snapshotChannel("aeron:ipc?alias=snapshot|mtu=65504");

Bench, reference engine, drain-only reader:

Orders	Take, default MTU (16/msg)	Take, 64 KB MTU (~800/msg)	Messages	Wire bytes
1M	218 ms	183 ms	62,503 → 1,257	86.0 → 82.2 MB
2M	449 ms	402 ms	125,003 → 2,510	172.0 → 164.3 MB
4M	753 ms	704 ms	250,003 → 5,016	344.0 → 328.5 MB

Live at 2M orders: take 366 → 332 ms, trigger to recording log 536 → 495 ms. The large-MTU recording replays correctly through the Archive.

What the bigger MTU saves:

• 50× fewer tryClaim/commit cycles (position-limit read, frame-header write, ordered commit).
• 50× fewer SBE message-header and group-header encodes and batch re-wraps.
• About 4.5% of the bytes: frame headers and slack in partial batches.

Why only about 10%: the remaining ~400 ms (bench, G1) is work every order needs regardless of frame size:

• Pointer-chasing the TreeSet across 2M separately allocated objects: ~110 ms.
• 13 field stores per order.
• First-touch page faults on the freshly mapped term buffer, because each snapshot gets a new publication.

Past this floor, a single thread needs a flatter book layout or a more compact record; transport tuning has little left to win. §9 gets under it another way, by walking the book on several threads.

Find the per-item floor first

Batching into fewer messages helps less than you expect once the per-message cost is already cheap. Measure the per-item floor (here: walk + encode + first-touch) before you tune transport, so you know what transport can still win.

6. Measurement traps that cost the most

• A contended box. One bench run coincided with a load average of 348 on 8 cores and reported serialize at 11 µs per order instead of 0.5 µs. A 353 s CPU stall even tripped the Aeron client’s ConductorServiceTimeoutException and aborted the run. The bench now prints the load average with every report and sets long driver/client liveness timeouts. Treat any timing taken at load > CPU count as an upper bound. On the shared host, the same noise later hid a 60% effect from one JVM flag (§8).
• Measuring restore and calling it take. The first zero-copy bench reported 2.25 s for the take at 2M orders, against 366 ms live. Its frames were consumed by the restore loader, which does TreeSet inserts at about 1.4 µs per order. Once the snapshot outgrew the term window, the publisher was back-pressured by restore speed. In the cluster the consumer is the Archive recorder, which only copies bytes. Time the take against a drain-only reader, and time restore in a separate pass.
• Bench vs single-node live. The bench ran slower than the single-node cluster: 402 vs 332 ms at 2M orders with a 64 KB MTU. Neither is what a client sees. The single-node runs wrote the Archive to tmpfs, which hides the fsync bound that §10 found on gp3. Use the bench for scaling shape and the §10 client pause for the production claim.
• Too few samples. With 15 takes per point, the 10K linear median carried warm-up noise and showed parallel encode 1.6× faster. With 1,200 takes the two tie (§9).
• The wrong collector. The test JVM defaulted to G1 while production runs ZGC, and on this walk ZGC is about 2× slower. Every number in sections 1–7 is a G1 number. See §8.
• Flags that never arrived. -D flags reach Gradle’s forked test JVM only if the build forwards them. An unforwarded -Dbench.snapshotChannel silently ran the default channel, and so did an empty "" argument in a shell loop. Check the configuration line the run printed, not the command you typed.

See Reading the Aeron Benchmark Honestly for the general version of these traps.

7. Scaling: heap, wire and take by book size

Reference engine. The first version of this table was a G1 linear fit over 200k–4M orders on the shared box. Every cell has since been measured except the G1 take at 10M.

Orders / shard	Heap, ZGC	Heap, G1	Wire	Linear take, G1	Linear take, ZGC	×8 take, ZGC
1M	245 MB	209 MB	82.2 MB	183 ms¹	298 ms	46.5 ms
5M	1,193 MB	1,012 MB	410.6 MB	650 ms	1,486 ms	179 ms
10M	2,382 MB	2,006 MB	821.3 MB	not measured	2,981 ms	361 ms

¹ Shared 8-vCPU box, 64 KB MTU (§5). Everything else is from quiet c7i.4xlarge instances: takes are medians (§8 for 5M linear, §9 otherwise); heap is used heap after 5 forced GCs, book minus empty engine, with -Xmx20g.

• ZGC needs ~18% more heap per order: 238–245 B against 201–209 B on G1, because it can’t use compressed oops (§8). Heap covers the order object, its TreeSet node, the by-id HashMap node and boxed key, and amortised table slots. The old G1 projections of ~1.0 GB at 5M and ~2.0 GB at 10M held.
• Wire size matched the fit. It stays at ~82 B per order from 1M to 10M.
• The G1 fit’s take projections don’t carry over. The fit gave ~0.9 s at 5M on the shared box; the quiet box measured 650 ms on G1 and 1,486 ms on ZGC. Take per order also isn’t constant: on ZGC it rose from ~82 ns at 100K to ~300 ns from 500K up (§9).
• None of these are client pauses. For what a client waits at production durability, see §10.

8. The production collector changes the picture

Everything above ran on G1. Production runs the engine with -XX:+UseZGC. Re-measured on a quiet, dedicated c7i.4xlarge (Xeon Platinum 8488C, 8 cores / 16 vCPUs, Corretto 25.0.4, load ≤ 2.9): linear take at 5M orders, median of 15 interleaved takes.

Config	ZGC	G1
Baseline	1,486 ms	650 ms
Agrona bounds checks off (-Dagrona.disable.bounds.checks=true)	927 ms (1.60×)	439 ms (1.48×)
Transparent huge pages for the heap	1,492 ms (1.00×)	564 ms (1.15×)
4 MB snapshot term (page faults per take 98k → 6k)	1,423 ms (1.04×)	565 ms (1.15×)
All three	855 ms (1.74×)	377 ms (1.73×)

• The take is memory-latency bound. A JFR profile on ZGC put ~67% of the take in walking the TreeSet and loading each order and ~27% in the stores. Claim and SBE overhead was at noise level.
• Page faults were not the bottleneck. Cutting them 16× with a 4 MB term saved ~4%.
• The contended box hid a 60% effect. The bounds-checks flag measured 0.96× at load 8 on the shared host and 1.60× on the quiet one.

Why ZGC is ~2.3× slower on this walk (JDK 25):

• ZGC can’t use compressed oops. It keeps GC metadata in the 64-bit reference (colored pointers, JEP 333), so -XX:+UseCompressedOops is silently overridden to false. HotSpot’s zArguments.cpp says “CompressedOops not supported”.
• Object size explains only part of it. Losing compressed oops costs ~15% in object size (183 vs 160 B per TreeMap entry; ~18% more heap per order across the whole book, §7), but size explains only ~15–30% of the gap. At equal size (G1 with compressed oops off), G1 was still ~1.5–1.75× faster. The rest is ZGC’s load barriers and/or post-compaction object layout. That test couldn’t separate the two and ran on the contended host, so treat this attribution as indicative.
• Compact object headers don’t help the walk. -XX:+UseCompactObjectHeaders (a product flag in JDK 25, accepted by ZGC) saves ~16 B per object but didn’t speed up the walk.

Measure on the collector you deploy

The 332 ms “live” result was G1, the test JVM’s default. Production’s ZGC was ~2× slower on the same code. If your benchmark harness doesn’t pass the production JVM flags, it isn’t measuring production.

9. Parallel encode: the book is frozen, so use the idle cores

While services take a snapshot, the cluster applies no commands, so the book is read-only and other threads may walk it. The design is off by default and enabled by a thread-count setting:

1. Partition. The matcher hands the publisher ordered partitions. The reference engine cuts each side’s red-black tree at a depth that gives ~16K-order subtrees, plus the cut-path nodes between them. Read in order, those are exactly the book’s order. That costs O(2^depth) node visits and nothing per order. The array-backed engine slices its lists by index.
2. Encode. A fixed pool of daemon workers writes each partition’s orders as raw 82-byte MeSnapshotOrders group entries (at the generated SBE offsets) into pooled off-heap buffers.
3. Stitch. The service thread waits for each partition in order and copies its entries into frames claimed exactly as the linear publisher claims them.

The output is therefore byte-identical to the linear snapshot. CI checks that byte for byte for both engines, at batch-boundary sizes and with every record type. A worker failure fails the snapshot.

Reaching the tree nodes needs --add-opens java.base/java.util=ALL-UNNAMED, because the partitioner reads TreeMap internals. Without the flag the engine logs once and stays linear: slower, never wrong.

Linear vs parallel, bench (64 KB-MTU IPC snapshot channel, drain-only reader), ZGC with production flags and no other tuning, quiet c7i.4xlarge. Median of 15 takes (5 runs × 3 rounds, interleaved) in ms, with the observed linear range. All 420 takes were byte-identical to linear.

Orders	Linear	×2	×4	×8	Best
10K	1.15 [0.74–1.35]	0.74	0.74	0.74	tie³
100K	8.18 [7.7–9.2]	4.40	4.37	4.38	1.9×
500K	139 [132–147]	53.1	31.4	25.0	5.6×
1M	298 [295–302]	116	63.2	46.5	6.4×
2M	608 [607–613]	232	122	87.5	7.0×
5M	1,490 [1,476–1,493]	585	298	179	8.3×
10M	2,981 [2,941–2,997]	1,199	617	361	8.3×

³ These 15-take medians suggest 1.6×, but that was warm-up noise. The 1,200-take tail run below shows a tie.

• Parallel ties at 10K and wins from 25K up. With 15 takes per point, parallel looked 1.6× faster at 10K, but the 15-take linear median carried warm-up noise. With 1,200 takes the two tie: p50 0.66 ms linear vs 0.67–0.69 ms parallel. From 100K up, parallel wins at every percentile; the crossover run below covers 25K–100K.
• Below 100K, ×2 is as good as ×8. At ~16K orders per partition a small book splits into only a few partitions (below ~32K, one per side), so threads beyond 2 have nothing to do. Those takes are under 10 ms either way.
• Large books gain most. Linear goes from ~82 ns per order at 100K, where the book fits in the 105 MB L3, to ~300 ns from 500K up, where the walk waits on memory. That is where several walkers in flight pay off: 8 threads reach ~36–46 ns per order.
• Two threads give ~2.5×, not 2×. The workers’ raw stores skip the SBE flyweight’s bounds checks, and the stitch overlaps the encode.
• Targets met under ZGC. 5M orders came in under a 500 ms target with 4 threads (298 ms) and with 8 (179 ms), and 10M did too with 8 threads (361 ms).
• G1 is faster here too. An earlier prototype of the same idea, with the §8 knobs on for both collectors, measured G1 about 2–2.7× faster than ZGC across 1–12 threads.

Where parallel starts to pay

Same machine and flags, 3 rounds × 400 takes = 1,200 per point, all byte-identical to linear, load ≤ 2.53. p50 in ms; speedup is linear ÷ ×2.

Orders	Linear	×2	×8	Speedup (p50)	p99, linear → ×2
10K	0.661	0.653	0.662	tie	0.91 → 0.86
25K	1.553	1.263	1.251	1.23×	1.74 → 1.46
50K	3.110	2.354	2.344	1.32×	3.77 → 2.68
75K	4.715	2.850	2.875	1.65×	5.05 → 3.29
100K	7.490	3.982	4.092	1.88×	8.91 → 4.43

Below 100K the absolute saving is under 4 ms. Sizes between 10K and 25K were not measured.

Tail latency of the take

The median says what a typical snapshot costs; the p99 says what the unlucky one costs, and a snapshot pause is a p99 event for every order that queues behind it. Same machine and flags, 3 interleaved rounds, with takes per point scaled to book size: 22,080 takes, all byte-identical to linear, load ≤ 4.8. Nearest-rank percentiles over all rounds. p999 is shown only where n ≥ 1,000, p99 only where n ≥ 100; otherwise ”—”. Times in ms.

Orders	Mode	n	p50	p99	p999	max
10K	linear	1,200	0.66	0.83	1.26	1.29
10K	×2	1,200	0.69	0.90	0.95	0.99
10K	×8	1,200	0.67	0.89	1.40	1.57
100K	linear	1,200	7.47	8.72	9.62	11.8
100K	×8	1,200	4.32	4.89	5.27	5.47
500K	linear	1,200	132	140	144	146
500K	×4	1,200	31.0	35.5	36.8	36.8
500K	×8	1,200	23.4	25.9	27.0	27.1
1M	linear	1,200	297	304	306	306
1M	×4	1,200	60.1	67.4	70.0	73.2
1M	×8	1,200	44.8	49.8	53.3	56.9
2M	linear	450	611	619	—	622
2M	×4	450	120	128	—	132
2M	×8	450	80.0	109	—	115
5M	linear	180	1,475	1,505	—	1,509
5M	×4	180	298	309	—	310
5M	×8	180	180	196	—	196
10M	linear	90	2,985	—	—	3,119
10M	×4	90	610	—	—	630
10M	×8	90	360	—	—	382

• A tight distribution, not a spiky one. From 500K up, max/p50 is 1.02–1.3 in every cell except 2M ×8 (1.44). The small books spread wider in relative terms, up to 1.90 at 100K and 2.35 at 10K, but every take there is under 12 ms. There is no GC or page-fault cliff.
• The 5M < 500 ms target holds at the tail: p99 309 ms with 4 threads and 196 ms with 8; max 310 and 196 ms.
• 8 threads have a slightly wider tail than 4 at 2M (p99/p50 1.37 vs 1.07). Eight busy workers on hyperthreads share cores with the driver and reader threads.

Use it for books of 25K resting orders or more, wherever cores are free during a snapshot: 8 threads on a 16-vCPU host, 4 where about 4 cores are free. In a real cluster it only pays once the Archive is no longer the bound (§10). The 25K threshold is the smallest measured size where parallel wins; at 10K the two tie, and 10K–25K was not measured. Keep linear when there are no spare cores, because the workers would compete with Aeron’s own threads, or when the java.util opens flag is missing. It stays off by default because spare cores are a deployment fact. The cost is memory: the pooled buffers keep about one snapshot’s order bytes resident between snapshots, ~410 MB at 5M orders.

Single-threaded means single writer, not single reader

A single-threaded state machine constrains mutation, not reads of frozen state. Aeron Cluster stops sequencing while services snapshot, so for that window the book is immutable. The cheapest speedup was the cores that were already idle.

10. End to end on a 3-node cluster: the Archive’s fsync sets the pause

Setup. 3 voters on c6in.16xlarge (64 vCPU, one cluster placement group) plus a c6in.4xlarge load generator. Production node configuration: ZGC, DEDICATED threading, FUNCTIONAL profile, file sync level 2, and the cluster directory (Archive, cluster log and media-driver directory) on an XFS-formatted gp3 EBS volume.

Measurement. The load generator preloads N resting orders through the real cluster ingress, then sends a fixed-rate probe stream: 5,000 IOC orders/s that cross nothing, each answered by one Release. Latency is measured from the scheduled send time, so probes the client could not even send during a pause are charged the pause (coordinated-omission corrected). Three snapshots per boot are triggered on the leader with ClusterTool. The client pause is the worst probe latency in the window around each trigger. Every scenario in the table sent 285,000 probes and lost none, and every snapshot’s stateHash was equal on all 3 nodes.

Ranges are over the warm snapshots (#2–3 of each boot):

Book	Encode	Disk (gp3)	Sync level	Archive block	ME take	Client pause	Normal client p50
1M	×8	125 MB/s	2	1 MB (Aeron default)	241–270 ms	457–474 ms	4.3 ms
5M	linear	125 MB/s	2	1 MB	~1.94 s	2.43–2.60 s	4.3 ms
5M	×8	125 MB/s	2	1 MB	~1.93 s	2.45–2.60 s	4.3 ms
5M	×8	1,000 MB/s	2	1 MB	~1.9 s	2.37–2.43 s	4.4 ms
5M	×8	1,000 MB/s (driver dir on /dev/shm)	2	1 MB	~1.9 s	2.38–2.43 s	4.3 ms
5M	×8	125 MB/s	2	16 MB	~1.95 s	2.43–2.58 s	4.2 ms
5M	×8	1,000 MB/s	2	16 MB	226–256 ms	747–847 ms	4.3 ms
5M	×8	1,000 MB/s	2	16 MB, code default	222–261 ms	747–763 ms	4.1 ms
5M	linear	1,000 MB/s	2	16 MB, code default	774–826 ms	1.28–1.30 s	4.1 ms
5M	×8	1,000 MB/s	1	1 MB	~1.9 s	2.38–2.42 s	3.6 ms
5M	linear	1,000 MB/s	0	1 MB	544–602 ms	940–986 ms	0.4 ms
5M	×8	1,000 MB/s	0	1 MB	90–108 ms	403–411 ms	0.3 ms
5M	×8	125 MB/s	0	16 MB, code default	85–149 ms	396–401 ms	0.3 ms
10M	×8	125 MB/s	2	1 MB	~5.1 s	5.9–6.1 s	4.8 ms
10M	×8	1,000 MB/s (driver dir on /dev/shm)	2	1 MB	~3.8–4.0 s	4.73–4.77 s	4.5 ms
10M	×8	125 MB/s	2	16 MB	~5.0 s	6.0 s	4.9 ms
10M	×8	1,000 MB/s	2	16 MB	464–496 ms	1.34–1.36 s	4.4 ms
10M	×8	1,000 MB/s	2	16 MB, code default	450–533 ms	1.33–1.40 s	4.2 ms

Rows are grouped by book size, then sync level and block size. Bold rows are the production claim. Rows marked “code default” ran with no -D flag, with the 16 MB block as the engine’s code default. The two ×8 ones at 1,000 MB/s reproduce the earlier runs that set 16 MB with the flag; the linear one at 1,000 MB/s and the sync-level-0 one on 125 MB/s are new tests.

• At sync level > 0, the Archive’s per-block fsync bounds the snapshot. Aeron’s RecordingWriter.onBlock calls FileChannel.force() after every block, and the block size (aeron.archive.file.io.max.length) defaults to 1 MB. The snapshot recording is written the same way: ~410 MB at 5M orders is ~410 syncs at ~4.7 ms each on gp3, about 1.9 s. That is why linear equals ×8, and 125 MB/s equals 1,000 MB/s, at sync level 2. At sync level 0 the bench’s parallel gain comes back inside the cluster: a ~100 ms take against ~570 ms linear.
• A 16 MB Archive block keeps full sync-level-2 durability and removes most of the syncs, if the disk can absorb the bytes. On gp3 at 1,000 MB/s the 5M take went from 1.9 s to 0.24 s and the client pause from 2.4 s to 0.8 s; the 10M pause went from ~4.7 s to 1.35 s. (That 10M baseline ran with the driver directory on /dev/shm, which made no measurable difference at 5M.) On gp3 at its 125 MB/s baseline it changed nothing: the bottleneck moved to disk throughput (~200 MB/s observed), still ~2 s at 5M.
• The client pause tracks the Archive finishing, not the engine’s take. Even at sync level 0 the cluster adds ~300–500 ms beyond the take: the consensus module’s snapshot, stopping the recording, and writing the recording log. Parallel encode can’t remove that floor.
• The same fsync sets everyday latency and throughput. Client p50 was 4.3 ms at sync level 2 against 0.3 ms at level 0, and preloading ran at 60K against 464K orders/s through 3-node consensus.
• The media-driver directory on EBS cost nothing measurable compared with /dev/shm.
• The first snapshot after a boot is usually the slow one. The gap is clearest where the take is fast: 1M orders on 125 MB/s took 1,158 ms against ~460 ms warm. In the linear runs bound by sync level 2, snapshot #1 was no slower, or barely slower, than the warm ones.
• Once the Archive block is fixed, parallel encode pays at production durability too. At 5M orders, sync level 2, 16 MB block, gp3 at 1,000 MB/s: the take was ~0.24 s with 8 threads against ~0.8 s linear (3.4×), and the client pause ~0.75 s against ~1.29 s (1.7×).
• At sync level 0, disk speed drops off the snapshot’s critical path. 5M ×8 on gp3 at 125 MB/s paused clients 396–401 ms, the same as at 1,000 MB/s (403–411 ms). The Archive writes into the page cache, and iostat showed the disk writes landing 7–13 s after the snapshot, during kernel writeback. Disk speed still matters for restart replay, for the long-run write rate, and once dirty pages exceed dirty_ratio (~20% of RAM).

Sync level 0 is a durability policy, not a tuning knob

At sync level 0 an acknowledged commit lives only in the page cache of a majority until the kernel writes it back. A single node crash loses nothing: the page cache survives a process crash, and the other two nodes have the data. A correlated power or OS loss of a majority within the ~30 s writeback window (dirty_expire_centisecs) loses acknowledged trades, and in this deployment all 3 voters share one placement group by design. Sync level 0 would cut the 5M pause to ~0.4 s and everyday p50 from 4.3 ms to 0.3 ms, but choosing it is a decision about what a crash may lose. This deployment chose sync level 2.

Recommendations from this test:

• Use a 16 MB Archive block. It is now the default in the engine’s cluster-node and backup-node code on the engine’s main branch, though not yet redeployed anywhere; an explicit -Daeron.archive.file.io.max.length still wins. Elsewhere, set archiveCtx.fileIoMaxLength(16 MiB) on the Archive context, or -Daeron.archive.file.io.max.length=16777216.
• Provision gp3 throughput for the Archive volume. The deployment’s Terraform set IOPS (6,000) but left throughput at the 125 MB/s baseline; 1,000 MB/s was the tested value. io2 is the alternative.
• Enable parallel encode once the Archive is no longer the bound. With the 16 MB block on 1,000 MB/s gp3 at sync level 2 it gave a 1.7× shorter client pause at 5M. Behind a 1 MB-block fsync it changes nothing.

The test also surfaced three production risks:

• Depth publishing caps throughput when the top of the book is crowded. The §2 fix stops the walk after 10 levels, but after every command the engine still re-walks every order in those levels. With ~500 orders per level the 3-node cluster managed only ~10K commands/s. The fix would be per-level totals maintained incrementally.
• ClusterTool needs Agrona 2.x’s --add-exports java.base/jdk.internal.misc, like every other JVM in the deployment.
• A client that only polls while the cluster works through a backlog gets timed out (10 s session timeout). It must send keepalives.

A faster take can’t fix a slow Archive

At file sync level 2 with Aeron’s default 1 MB Archive block, linear and 8-thread parallel encode paused clients for the same ~2.4 s at 5M orders, on 125 MB/s and 1,000 MB/s gp3 alike. Measure the client pause on the production durability setting before spending effort on the take.

11. Follower restart: the snapshot is the short part of recovery

The snapshot costs time a second way: a node that restarts has to load it before it can rejoin. To measure that without an election in the number, the leader stayed put and only followers were restarted. A leader crash would add the 10 s leader-heartbeat timeout (Aeron’s default, not overridden here) and an election before any of this starts.

Setup. The §10 cluster with the production configuration: ZGC, sync level 2, gp3 at 1,000 MB/s, the 16 MB Archive block, and 8-thread parallel encode, with -Xmx24g and no heap pre-touch. One boot per run: a snapshot at +10 s, then three kill -9 / relaunch cycles of a follower (killed at +25, +85 and +145 s, relaunched 15 s later, alternating the two followers), and a final snapshot at +215 s. The same coordinated-omission-corrected probe stream recorded every latency, and a counters watcher timestamped each restarted node’s election-state transitions. In every run the leader never changed or re-elected (its election state stayed CLOSED), no probe was lost, and the final stateHash was equal on all 3 nodes.

Book	Probe rate	Launch → election starts	In-node snapshot restore	Local log replay	Catch-up to CLOSED	Launch → rejoined
5M	5,000/s	4.05–4.20 s	2.84–3.01 s	31 ms–1.64 s	49.8–111 s; first restart never closed within 105 s	56–116 s
10M	5,000/s	7.29–7.58 s	5.80–6.10 s	41 ms–1.69 s	47–61 s; first restart never closed within 105 s	56–69 s
5M	500/s	4.04–4.23 s	2.88–3.05 s	31–41 ms	0.20–0.44 s	4.55–4.64 s

The 5M row at 5,000/s combines two runs (six restarts); the others are one run of three restarts each. The log replayed since the snapshot was 33, 100 and 167 MB at 5,000/s (3.3, 10 and 16.7 MB at 500/s), written 29, 89 and 149 s after the snapshot.

• The snapshot’s share of recovery is ~4 s at 5M and ~7 s at 10M. That is ~0.9 s of JVM and driver start plus the in-node restore of 2.9 s and 5.9 s (≈ 0.58 µs per order, derived), in line with the bench restore. Local log replay is fast: ~167 MB replayed in 1.3–1.7 s.
• Under traffic, finishing catch-up dominates, and the data isn’t the problem. 10–15 s after relaunch the follower’s own Archive recording (rec-pos) was within ~0.1–0.3 MB of the leader’s commit position, a sampling skew of ~0.1–0.3 s of traffic. Yet its election stayed in FOLLOWER_CATCHUP for another 40–100+ s, and in the first restart of every 5,000/s run it never finished before that node was killed again. At 500/s catch-up finished in under 0.5 s every time. How long catch-up takes to conclude depends on live traffic; the mechanism inside Aeron is not confirmed.
• A follower’s Cluster commit-pos counter is the leader’s commit position, as sent to it, not the follower’s own progress. Watch its Archive rec-pos instead.

Client impact, from every probe at 5,000/s unless noted:

Book	Window	p50	p99	max	Lost
5M	all 3 up	4.05–4.07 ms	6.18–6.23 ms	9.6–10.9 ms	0
5M	one follower down (15 s)	4.52–4.83 ms	7.17–7.79 ms	9.5–33.3 ms	0
5M	relaunch → rejoined (65.9 s)¹	4.33 ms	6.96 ms	15.4 ms	0
10M	all 3 up	4.13 ms	6.27 ms	8.0 ms	0
10M	one follower down (15 s)	4.74–4.87 ms	7.20–7.47 ms	9.8–12.7 ms	0
10M	relaunch → rejoined (69.2 s)¹	4.59 ms	7.15 ms	14.5 ms	0
5M at 500/s	all 3 up / one follower down	4.88 / 5.46–5.48 ms	8.90 / 9.56–9.72 ms	10.9 / 11.2–11.6 ms	0
5M at 500/s	relaunch → rejoined (4.5–4.6 s)	5.41–5.81 ms	9.74–10.04 ms	12.9–18.0 ms	0

¹ The other 5,000/s rejoin windows also contain the final snapshot at +215 s, so they measure the §10 snapshot pause, not the restart, and are left out. Windows for first restarts that never closed have no end.

With one follower down, every commit waits on the remaining follower: p50 rose ~0.5–0.7 ms and p99 ~1 ms. No probe was lost, and a follower restart never disconnected a client.

A restarted follower isn’t a spare voter until catch-up closes

Under 5,000 probes/s, the cluster ran without a spare voter for ~1–2 minutes after each follower restart, not for the ~4–7 s the snapshot restore took. A second node failure in that window would stop commits. Watch the restarted node’s election state reach CLOSED, not its log position, before you treat the cluster as back at full strength.

12. Still open

• Restore was not optimized. Through the production loader on the quiet box, linear restore took 520 ms at 1M (0.52 µs per order), 3.38 s at 5M (0.68 µs) and 7.55 s at 10M (0.76 µs) on ZGC, and almost the same on G1 (0.52 / 0.65 / 0.74 µs). It barely depends on the collector, and the per-order cost grows with book size: deeper trees, more cache misses. This is a bench restore from an IPC publication, with no disk read and no log replay; inside a restarting node it took 2.9 s at 5M and 5.9 s at 10M (§11). Each order costs a TreeSet insertion (~21 comparisons at 2M), plus a RestingOrder object, a tree node, a HashMap node and a boxed Long. Two likely wins, not yet measured: rebuild each TreeSet in one linear pass from the already-sorted input (TreeMap.buildFromSorted via a SortedSet source), and pre-size the id map.
• The OMS snapshot: the large-state bug is fixed, but it is not zero-copy. It used to be one offer() restored by polling exactly one fragment with no assembler, so a state larger than one frame logged “bad oms payload length” and restored empty. The 64 KB MTU only raised that ceiling from ~1.3 KB to ~64 KB. It now gets the same Begin / batches / End treatment: the state goes out in chunks of one claim each, and the loader runs from Begin to End, checks the counts, and throws rather than restore a partial state. Old single-message snapshots are rejected loudly, not decoded. What remains is that the OMS state image is still built as byte[]s before chunking.
• The single-thread floor. A linear take can’t go much below ~0.15 µs per order on G1, or ~0.3 µs on ZGC, without a flatter book (array- or off-heap-backed price levels) or a compact record format. Parallel encode (§9) gets under it without either.
• None of the §10 recommendations is running in production yet. The 16 MB Archive block (on by default) and parallel encode (off unless a thread count is set; threshold 25K) are merged to the engine’s main branch, but no environment has been redeployed with them. The gp3 throughput provisioning is a recommendation only: the Terraform is unchanged, and the sync level stays at 2.
• Per-command latency at file sync level 2 (~4 ms p50 on gp3) is a separate production question; the options named are io2, instance store plus replication, or group commit.
• The end-to-end results cover one cluster and one instance type, with 3 snapshots or 3 restarts per scenario. They show ranges, not percentiles.
• Follower catch-up under load is unexplained. It took ~1–2 minutes at 5,000 probes/s and under 0.5 s at 500/s (§11). The mechanism inside Aeron’s catch-up is not confirmed; catch-up termination under load and the LOW_LATENCY profile’s idle strategies are the open leads, neither tested.
• Leader-crash recovery was not measured. It adds the 10 s leader-heartbeat timeout and an election to everything in §11 (see Tuning Cluster Failover Time). In an earlier, abandoned leader-kill attempt, the crash closed the client’s session, so clients must be able to reconnect.
• The pause itself. Removing the cluster-wide pause entirely still needs a design change, such as incremental snapshots or snapshotting on a standby node. Tuning won’t get you there. See Cluster Standby and Multi-AZ HA Design.

Members only

The rest of this page is for approved members.

Still on this page:

1. The result
2. 1. One claim for the whole book caps you at 1,376 bytes
3. 2. Chunk first, and find the O(n²) hiding elsewhere
4. 3. The serializer: two bad constant factors and one bad complexity
5. 4. Aeron’s own pattern: Begin / batches / End, zero-copy, one pass
6. 5. A 64 KB snapshot MTU: 50× fewer messages, about 10% faster
7. 6. Measurement traps that cost the most
8. 7. Scaling: heap, wire and take by book size
9. 8. The production collector changes the picture
10. 9. Parallel encode: the book is frozen, so use the idle cores
11. 10. End to end on a 3-node cluster: the Archive’s fsync sets the pause
12. 11. Follower restart: the snapshot is the short part of recovery
13. 12. Still open

Log in or request access

Sign in with an emailed link, no password. New accounts are approved by hand.