Skip to content

12 Mistakes Teams Make Adopting Aeron®

The most expensive Aeron® mistakes are defaults left alone and one client-side omission. Stock Cluster settings take ~11 s to fail over after a leader dies (Aeron 1.48.11, bare metal). Stock NAK timers make every lost packet on a multicast or MDC stream cost 10–20 ms, however well your buffers are sized. And a cluster client that never polls egress stays back-pressured forever after a failover. None of these shows up in a quick functional test; all of them show up in production.

For how Aeron works internally, read The Aeron Files.

1. Leaving the NAK timers at their defaults on multicast and MDC

Section titled “1. Leaving the NAK timers at their defaults on multicast and MDC”

A receiver that detects a gap waits a randomised delay of up to aeron.nak.multicast.max.backoff (default 10 ms) before it NAKs. That window was sized for internet-scale RTTs; on a LAN it is absurdly oversized. It governs every multicast-semantics stream, including Aeron Cluster’s log-replication channel. Measured on bare metal (1 publisher → 3 MDC receivers, 224 B at 50k msg/s, seeded 0.01% loss), retuning the timers took p99.9 from 8,286 µs to 105 µs; p50 stayed at 37 µs.

Fix direction: derive the backoff from your RTT and loss budget, and set aeron.nak.multicast.group.size to the real receiver count. These are media-driver properties, so they must reach the driver process, not your application JVM. Deep dive: NAK Timer Tuning.

2. Raising so_rcvbuf without checking what the kernel actually granted

Section titled “2. Raising so_rcvbuf without checking what the kernel actually granted”

The driver refuses to start if the receiver window exceeds SO_RCVBUF, but it cannot enforce net.core.rmem_max. If the kernel ceiling is lower, the kernel silently caps the buffer and the driver only prints a warning. The ENA RX ring is a second, time-denominated trap: at 10 Gbps of ~1500-byte frames, the 1024-entry default survives only ≈ 1.2 ms of receiver stall, against the ≈ 10 ms a CFS wakeup delay can produce. Either way you get RcvbufErrors or rx_overruns, then gaps and NAKs.

Fix direction: size the four buffers bottom-up as one chain, ring to max first. Deep dive: The Four Receive-Path Buffers.

3. Firing unpaced bursts and looking for the latency at the receiver

Section titled “3. Firing unpaced bursts and looking for the latency at the receiver”

An offer() loop dispatches at memory speed; the media driver’s single Sender thread drains at a fixed rate. The difference waits in the sender’s own term buffer. In one AWS measurement (160-byte messages), an unpaced 90,000-message burst reached ≈ 55 ms of sender-side queue latency (worst observed ≈ 67 ms), with UDP drops, NAKs and back-pressure all at zero. Dashboards that watch only loss and back-pressure never see it.

Fix direction: measure your drain rate and pace bursts at the source against a latency target. Deep dive: Sender Queueing and Burst Pacing.

taskset places a thread; it does not keep other tasks off that core. isolcpus empties cores; on its own it just shrinks the machine. You need both, plus IRQ steering. In Mark Price’s measurements, affinity plus isolcpus took worst-case inter-thread latency from 11.5 ms to 14.8 µs; IRQ steering and nosoftlockup then cut residual jitter from 15 µs to 2.5 µs. Affinity without isolation keeps the p50 benefit and leaves the tail exposed.

Fix direction: boot parameters to carve out and de-noise, explicit placement onto the carved cores, IRQs steered to housekeeping cores. Deep dive: Core Isolation & Thread Pinning.

A single flow (one 5-tuple) is capped at 5 Gbps within a VPC and 10 Gbps within a cluster placement group. One Aeron stream pinned to one 5-tuple cannot exceed that, however wide the instance link is. Small, high-rate messages usually hit the instance PPS ceiling before bandwidth. If throughput plateaus while bandwidth utilisation stays low, you are packet-bound: only fewer, larger packets (batching, a bigger MTU) or a bigger instance moves it.

Fix direction: check PPS, per-flow, bandwidth, utilisation headroom and BDP before tuning Aeron. Deep dive: AWS Hard Limits You Cannot Tune Past.

6. Shipping the stock leader heartbeat timeout

Section titled “6. Shipping the stock leader heartbeat timeout”

aeron.cluster.leader.heartbeat.timeout defaults to 10 s, about 90% of the stock failover time. Measured on Aeron 1.48.11, bare metal, an unplanned leader death took ~11.0 s to recover: the 10 s detection plus ~1 s of election. A 1 s heartbeat timeout (with a 200 ms heartbeat interval and a 500 ms election timeout) gives about 1.5 s failover and still rides out ordinary GC and scheduling stalls. Going much lower trades recovery speed for false failovers on a healthy leader.

Fix direction: lower the timeout to what your worst realistic leader stall allows, keep the heartbeat interval well below it, and soak before going sub-second. Deep dive: Tuning Cluster Failover Time.

A client that offers ingress but never calls pollEgress() never sees the new-leader event. After a failover it stays back-pressured forever, whatever the timeouts are set to. This one is not a tuning problem, and it makes item 6 irrelevant until it is fixed.

Fix direction: poll egress and handle EgressListener.onNewLeader. Deep dive: Tuning Cluster Failover Time and Client-Cluster Communication.

8. Letting non-determinism into the clustered service

Section titled “8. Letting non-determinism into the clustered service”

Every node must produce bit-identical state from the same log. System.currentTimeMillis(), UUID.randomUUID(), HashMap iteration order, Math.sin() across JVMs and CPUs, and any HTTP or database call inside onSessionMessage all break that. An asymmetric onTakeSnapshot()/loadSnapshot() pair does the same through the snapshot. None of this moves p50, p99 or throughput. The cluster looks healthy until a failover promotes a leader whose state differs.

Fix direction: take time from cluster.time(), use ordered collections, StrictMath or fixed-point, push external data in through ingress, and checksum snapshots and compare state hashes across nodes. Deep dive: Failure-Mode Runbook: Snapshots and Determinism.

9. Tuning the snapshot encode while the Archive fsync sets the pause

Section titled “9. Tuning the snapshot encode while the Archive fsync sets the pause”

A ClusteredService snapshots on its single service thread, so the book cannot trade while onTakeSnapshot runs, and every order that arrives queues behind it. Two numbers measure that: the take (time the service thread spends writing) and the client pause (how long a client actually waits). Making the encode faster shortens the take. At production durability on a real 3-node cluster, it is not the take that sets the pause, so a faster encoder alone can leave clients waiting just as long.

Fix direction: measure the client pause, not just the take, and fix the Archive’s write path before the encoder. Deep dive: Aeron Cluster Snapshot Performance.

purgeSegments respects segment boundaries and active recordings. It does not know whether the bytes it deletes are the only path back for a stale node. Election catch-up transfers log, never the leader’s snapshot, so a node whose next position has been purged cannot rejoin; it has to be rebuilt. “Keep seven days” is a proxy for the positions that actually decide restart versus rebuild.

Fix direction: measure snapshot and log positions per node, pick a retention model deliberately, and validate before you delete. Deep dive: Archive Retention and Purge.

The Aeron benchmark harness fixes the rate and reports latency; it never searches for the knee, the rate where latency crosses your SLA. Its burst knob (batch.size) forces messages to coalesce into shared datagrams, which an evenly paced production feed does not do, so the same code on the same hardware can look healthy at a rate that saturates it in production.

Fix direction: sweep the rate yourself at your real message size and arrival shape, and quote the rate where p99.9 crosses your SLA. Deep dive: Reading the Aeron Benchmark Honestly.

12. Buying the DPDK driver as a blanket speed-up

Section titled “12. Buying the DPDK driver as a blanket speed-up”

The headline “DPDK is N× faster” figures compare a healthy DPDK point with a saturated kernel-driver point. Below the kernel drivers’ knee the median gap is small. DPDK also costs dedicated busy-spinning cores, reserved hugepages and a dedicated ENI, and a misconfigured bind degrades it silently, with no error.

Fix direction: adopt it for the packet-rate ceiling or a predictable tail, after measuring your per-operation service cost. Deep dive: When You Need the DPDK Media Driver.

This site is not affiliated with, endorsed by, or sponsored by Adaptive Financial Consulting Limited or the Aeron project. Aeron is a registered trademark of Adaptive Financial Consulting Limited.

Aeron is a trademark of Adaptive Financial Consulting Limited in the United Kingdom and other countries.