Monitoring Aeron®: Metrics and Alerting for Your Observability Platform

Aeron exposes a rich set of lock-free counters in shared memory — but it ships no /metrics endpoint. To get those signals onto your company’s Prometheus/Grafana/Datadog stack you either write a small exporter (OSS) or run Aeron Insights (Premium). This page covers both paths, then maps the counters that actually matter to the failure modes they predict and the alert conditions to set.

Note

This is the platform-integration and alerting page. For reading counters interactively at the console, the drafted aeron-stat and Cluster Diagnostic Counters pages go deeper on individual signals; for the protocol-level meaning of each counter, The Aeron Files has a dedicated section. Verified against Aeron OSS 1.51.1 and Aeron Insights 0.3.0-beta (2026).

How Aeron exposes telemetry: the CnC file

Every Aeron media driver maintains a command-and-control file, cnc.dat, memory-mapped in its Aeron directory. Among its regions are a counters metadata buffer, a counters values buffer, and an error-log buffer. Counters are plain 64-bit values updated lock-free by the driver; any external process can memory-map the file read-only and read them live, with — in Aeron’s own words — “no significant impact on performance.”

That is the whole integration surface. There is no push, no agent protocol, no HTTP — you map the file and iterate. The reference reader is AeronStat (the aeron-stat script), which loops once per second and prints every counter. Under the hood it does exactly what any exporter does:

// The canonical read pattern — Agrona's CountersReader over the mapped cnc.dat
final CountersReader counters = cncFileReader.countersReader();
counters.forEach((counterId, typeId, keyBuffer, label) -> {
    final long value = counters.getCounterValue(counterId);
    // → emit {counterId, typeId, label, value} to your metrics client
});

Counters are grouped by type id: 0–99 client/driver, 100–199 archive, 200–299 cluster. That type id is your routing key when you turn counters into metrics.

Key Insight

The CnC file is the API. Because reads are lock-free map-and-poll against shared memory, a scraper adds no back-pressure and no latency to the driver — you can poll it aggressively without touching your p99. This is why the DIY-exporter pattern below is cheap and safe, not a compromise.

Path 1 — OSS: write (or reuse) a counter exporter

Aeron OSS has no native Prometheus exporter. This is by design, and Aeron’s own cookbook points you at the DIY route: “Read the counter data using CountersReader” and feed a metrics system “such as DataDog or Prometheus.” The pattern is a per-node sidecar that maps that node’s cnc.dat, iterates counters on a timer, and exposes /metrics (Prometheus scrape) or pushes to StatsD/Datadog/CloudWatch.

You don’t have to write it from scratch — a couple of community exporters already implement this:

Exporter	What it is	State
singleplayer88/aeron-exporter	Reads cnc.dat via CncFileReader/CountersReader, serves /metrics (Jetty). Apache-2.0.	Recently maintained (2026)
fairtide/aeron-prometheus-stat	Same CnC-reader pattern, renames counters to Prometheus conventions; ships Docker Compose. Apache-2.0.	Older (~2023)
scalecube/scalecube-metrics	Telemetry library over Agrona counters + HdrHistogram, with a Prometheus-over-Aeron demo.	Actively maintained

Vet before you trust

These are community projects, not Aeron releases — pin a commit, read the code, and confirm it handles CnC version-major mismatches (the file format is versioned; a reader built against a different major throws on map). Treat any third-party exporter as sample code you own, not a supported dependency.

For Datadog, StatsD, or CloudWatch there is no off-the-shelf integration — the same CountersReader.forEach loop pushes to that platform’s client instead of exposing /metrics. A log-scraping fallback (parse the once-per-second aeron-stat console output) works but is strictly worse: coarser, lossy, and it re-parses text you already have as numbers.

Path 2 — Premium: Aeron Insights

Aeron Insights (Premium, currently beta) is the turnkey version of Path 1. It ships Prometheus exporters, a sample Grafana dashboard, and a CLI. The exporters “gather data by reading the files created by Aeron, principally the CnC file” plus the archive/cluster mark files — i.e. the same read-only approach, packaged and supported — in two per-node delivery models:

• HTTP-server exporter (default :8080, localhost) — Prometheus scrapes each node directly. This is the common approach.
• Push-client exporter — pushes to a Prometheus Push Gateway on an interval (default 1s).

The payoff is that Insights emits named, documented Prometheus metrics out of the box (cluster_*, driver_*, archive_*), with a Grafana “Cluster Overview” dashboard, instead of raw type-id’d counters you have to name and curate yourself.

	OSS DIY exporter	Premium Aeron Insights
Reads	cnc.dat (via your code)	cnc.dat + mark files (packaged)
Metric names	You define them	Documented cluster_* / driver_* / archive_*
Grafana dashboard	You build it	Sample “Cluster Overview” ships
Delivery	Whatever you write	HTTP scrape or Push Gateway
Support	Community / self	Adaptive (beta)
Cost	Free (Apache-2.0)	Subscription

Key Insight

The telemetry source is identical — both read the same CnC counters. What Premium sells here is the last mile: names, dashboards, packaging, and support. If you already run a Prometheus stack and have the engineering to own ~200 lines of exporter, OSS gets you the same signals. Insights is worth it when you want a supported, dashboard-ready path and don’t want to curate metric names across upgrades.

The counters that matter (and what bad looks like)

You do not need all of Aeron’s counters on a dashboard. For a clustered matching engine, these are the load-bearing ones. OSS names are the counter labels / type ids from the CnC file; the Insights column is the Prometheus metric name where one exists.

Signal	OSS counter (type id)	Insights metric	Read it as
Commit position	Cluster commit-pos: (203)	cluster_commit_position_total	The heartbeat of the cluster — quorum-replicated log position
Append position	Archive rec-pos of the log recording	(no dedicated metric)	Where the leader has appended; gap vs commit = follower lag
Node role	Cluster node role (201)	cluster_node_role	2 = Leader, 0 = Follower, 1 = Candidate
Consensus state	Consensus Module state (200)	cluster_consensus_module_state	1 = Active, 3 = taking a snapshot
Election state / count	Cluster election state (207) / Cluster election count (238)	cluster_election_state (count not exported)	Steady state = 17 (Closed); count should never climb
Leadership term id	Cluster leadership term id (239)	(not exported)	Increments once per legitimate leader change — faster than that is churn
CM cycle time	Cluster max cycle time in ns (216) / exceeded (217)	cluster_max_cycle_time_seconds, cluster_cycle_time_threshold_exceeded_total	Duty-cycle latency of the consensus module
Service cycle time	Cluster container max cycle time in ns (218) / exceeded (219)	cluster_service_max_cycle_time_seconds, cluster_service_cycle_time_threshold_exceeded_total	Duty-cycle latency of your service code
Driver cycle time	Conductor/Sender/Receiver max cycle time … in ns + work cycle exceeded threshold count (type 0)	driver_max_cycle_time_seconds{agent}, driver_cycle_time_threshold_exceeded_total{agent}	The layer under the cluster — first place lost core isolation shows up
Errors	Cluster Errors (212) / Cluster Container Errors (215)	cluster_errors_total, cluster_service_errors_total	Monotonic tally — any delta warrants a look
Timed-out clients	Cluster timed out client count (213)	cluster_timed_out_clients_total	Sessions dropped for inactivity — gateway stalling, or your session timeout is too tight
Valid snapshots	Cluster snapshot count (205)	cluster_valid_snapshots	Count of usable snapshots
Snapshot age	(derive from recording-log timestamp)	cluster_last_snapshot_timestamp_seconds	Old snapshot ⇒ long replay on recovery
Snapshot duration	Total max snapshot duration in ns (234) / exceeded (235); per-service (236/237)	cluster_max_snapshot_duration_seconds, cluster_snapshot_duration_threshold_exceeded_total	Long snapshots stall the hot path (OSS)
Back-pressure	snd-bpe (13) / Sender flow control limits	driver_sender_bpe_total (also per-streamId)	“Should be zero in a healthy system”
Loss / NAK	NAKs sent/received, Retransmits sent (type 0)	driver_nak_messages_*_total, driver_retransmits_sent_total	Network loss ⇒ replication drag
Loss gap fills	Loss gap fills (type 0)	driver_loss_gap_fills_total	Loss that was skipped, not recovered — only non-zero on non-reliable streams
Malformed / dropped	Invalid packets, Short sends (type 0)	driver_invalid_packets_total, driver_short_sends_total, driver_flow_control_over_runs_total	All expected to be zero; non-zero means MTU, socket-buffer, or a slow subscriber
Liveness	(freshness of counter updates)	cluster_activity_age_seconds, driver_heartbeat_age_seconds	Seconds since the component was last seen active

Two things people get wrong

“Appended position” is not a cluster counter — it’s the Aeron Archive rec-pos counter for the log recording. Compute the follower-lag gap as rec-pos(log) − commit-pos; neither OSS nor Insights gives you the gap pre-computed. And the error text lives in the distinct-error log (dumped point-in-time by ClusterTool errors), not in a metric — the Cluster Errors counter is only a count. Alert on the count; page a human to read the text.

Insights does not export election count — and that changes your alerting

Cluster election count (238) and Cluster leadership term id (239) exist as OSS counters but have no documented Aeron Insights metric. Insights gives you cluster_election_state — a stage gauge that is 17 (Closed) at rest and transient otherwise. A Prometheus scrape at 1–15 s will routinely miss a whole election, because the default election timeout is 1 s: the state gauge can leave and return to 17 entirely between two scrapes, and you see nothing.

So “alert on rising election count”, the single most useful cluster-stability rule, is not available on Insights alone. Either scrape counter 238 with your own exporter alongside Insights, or fall back to counting leader transitions (changes(cluster_node_role[…])), which is coarser and misses elections that re-elect the same leader. This is the one place where the OSS DIY path gives you a signal the Premium product doesn’t.

Consensus-module state: which values page you

Consensus Module state (200) / cluster_consensus_module_state is an enum, and only two of its seven values are benign. Alerting on ”!= 1” will page you during every snapshot.

Value	State	Treat as
0	INIT	Normal at startup and during recovery. Page if it lingers past your expected replay time — a node stuck replaying a long log is a node not in the quorum.
1	ACTIVE	Healthy steady state.
2	SUSPENDED	Someone toggled the cluster via ClusterControl/ClusterTool. Ingress is not being processed. Page — this is almost never intentional in production.
3	SNAPSHOT	Normal, and expected to be brief. Sustained ⇒ see the snapshot-duration alert.
4	QUITTING	Node is leaving. Expected during a planned stop, otherwise page.
5	TERMINATING	Shutting down — including the involuntary ClusterTerminationException path (e.g. archive storage exhaustion). Page.
6	CLOSED	Terminal. The node is gone; liveness/heartbeat-age will confirm.

The practical rule: page on 2, 4, 5, 6 immediately; page on 0 only after a startup grace period sized to your log-replay time; never page on 3 (alert on its duration instead).

Alerting: mapping metrics to failure modes

Each of these ties a metric to a failure-mode runbook scenario. Aeron publishes almost no numeric thresholds — the timers below are documented; the rest are operator judgment, so calibrate against your own baseline and start loose to avoid alert fatigue.

Alert	Condition	Why / documented basis
Commit stall (Critical)	commit-pos delta == 0 while state == Active for > ~10 s	Aeron’s own “No catchup progress” warning fires when commit doesn’t advance for longer than the leader heartbeat timeout (default 10 s). Causes: slow app code, machine load, GC.
No leader (Critical)	No node reports role == Leader (2) for > election timeout	Election in progress = cluster not committing.
Election churn (High)	election count (238) increasing, or election state not returning to 17 (Closed)	Any increment is a p99 cliff. Triggers: heartbeat timeout, lost quorum. (“Election storm” is not a documented term — set your own rate.)
Leadership churn (High)	leadership term id (239) incrementing more than your planned failover rate	Term id advances once per legitimate leader change; a climbing term with no maintenance underway is the cluster re-electing itself.
Clients timing out (Medium)	cluster_timed_out_clients_total delta > 0	Sessions dropped for inactivity — a gateway/OMS that stopped polling, or a session timeout tighter than your client’s worst-case pause. Distinguish from driver_client_timeouts_total, which is media-driver clients, not cluster sessions.
Follower lag (Medium)	rec-pos(log) − commit-pos growing over time	Leader appending faster than a quorum acks — heading toward back-pressure or quorum loss. No documented gap threshold.
Cycle-time breach (Medium)	cycle time exceeded counters (217/219) incrementing	Duty-cycle threshold default is 100 ms for CM and service. Spikes = GC / CPU starvation; a 10 ms cycle is a 10 ms hole in p99.
Driver cycle-time breach (Medium)	driver_cycle_time_threshold_exceeded_total{agent} incrementing	Same 100 ms default for the driver’s conductor, sender, and receiver. This fires before the cluster counters do when the cause is shared-core contention or lost isolcpus — the driver threads feel it first. See core isolation and pinning.
Errors rising (High)	Cluster Errors (212) delta > 0	Then read the text via ClusterTool errors. No numeric rate documented.
Snapshot too old (Medium)	now − cluster_last_snapshot_timestamp_seconds > your recovery budget	Snapshots are manual in Aeron — an old snapshot means a long log replay on restart. See Snapshot Failure and Recovery.
Snapshot too slow (Low/Medium)	snapshot-duration exceeded (235) incrementing	Threshold default 1 s. In OSS the snapshot stalls the hot path; on the leader that lands on max latency.
Back-pressure / loss (Medium)	snd-bpe, NAK, or retransmit rate rising above baseline	Subscribers/followers not keeping up, or network loss dragging replication. No documented rate threshold.
Silent data skip (High)	driver_loss_gap_fills_total delta > 0 on a cluster stream	A gap fill means loss was skipped rather than recovered. It only happens on non-reliable streams, so on cluster traffic it means someone configured reliable=false on a stream that must not lose data. Investigate the config, not the network.
Malformed frames (Medium)	driver_invalid_packets_total, driver_short_sends_total, or driver_flow_control_over_runs_total delta > 0	All three are documented as “expected to be zero”. Invalid packets ⇒ MTU/path problem or a foreign sender on the port; short sends ⇒ socket buffer too small; over-runs ⇒ one receiver falling behind the rest of the stream.
Storage exhaustion (Critical)	Free space on archive/cluster volumes below your floor	Archive storage exhaustion throws ClusterTerminationException and terminates the node — the one snapshot/archive failure that is always terminal. Alert before it happens.
Node dark (Critical)	cluster_activity_age_seconds / driver_heartbeat_age_seconds climbing, or scrape target down	The component stopped updating counters — process dead or wedged.

The four timers are different — don’t conflate them

Leader heartbeat timeout = 10 s. Election timeout = 1 s. Duty-cycle threshold = 100 ms. Snapshot-duration threshold = 1 s. These are separate knobs at separate layers; an alert that confuses them (e.g. treating a 100 ms cycle breach as a 10 s stall) will either page constantly or miss the real outage. Insights’ processing-time metrics default to a 1 s threshold, distinct from the OSS 100 ms duty-cycle default — be explicit about which layer a threshold belongs to.

Alert rules you can paste

Prose conditions are where alerting goes to die, so here are the same rules as PromQL. They use the Insights metric names; on a DIY exporter substitute your own, and note that commit_position_total is a position, not a counter — rate() on it is bytes/s of committed log, which is exactly what you want, but increase() resets look like restarts.

Every cluster metric carries a clusterId label (services additionally serviceId), so keep by (instance, clusterId) on anything you want to attribute to a node.

groups:
- name: aeron-cluster-stability
  rules:

  # 1. Commit stall — the cluster is up but not committing.
  #    Window > leader heartbeat timeout (10s) so a normal election doesn't fire it.
  - alert: AeronCommitStalled
    expr: |
      rate(cluster_commit_position_total[1m]) == 0
        and on(instance, clusterId) cluster_consensus_module_state == 1
        and on(instance, clusterId) cluster_node_role == 2
    for: 15s
    labels: { severity: critical }
    annotations:
      summary: "Leader ACTIVE but commit position flat for 15s"

  # 2. No leader anywhere in the cluster.
  - alert: AeronNoLeader
    expr: sum by (clusterId) (cluster_node_role == 2) == 0
    for: 5s
    labels: { severity: critical }

  # 2b. Split brain: more than one node claims leadership.
  - alert: AeronMultipleLeaders
    expr: sum by (clusterId) (cluster_node_role == 2) > 1
    for: 5s
    labels: { severity: critical }

  # 3. Leadership churn. Insights exports no election-count metric (see the callout above),
  #    so count role transitions. Add a rule on counter 238 if you scrape it yourself.
  - alert: AeronLeadershipChurn
    expr: sum by (clusterId) (changes(cluster_node_role[15m])) > 2
    labels: { severity: warning }
    annotations:
      summary: "More than one leader change in 15m"

  # 3b. Election not settling: state stuck away from 17 (CLOSED).
  - alert: AeronElectionNotSettling
    expr: cluster_consensus_module_state == 1 and cluster_election_state != 17
    for: 30s
    labels: { severity: warning }

  # 4. Consensus module in a state that needs a human.
  #    2=SUSPENDED 4=QUITTING 5=TERMINATING 6=CLOSED. 3=SNAPSHOT is benign.
  - alert: AeronConsensusModuleBadState
    expr: cluster_consensus_module_state == 2 or cluster_consensus_module_state >= 4
    for: 10s
    labels: { severity: critical }

  #    0=INIT is only bad if it persists — size `for` to your log-replay time.
  - alert: AeronStuckInitialising
    expr: cluster_consensus_module_state == 0
    for: 10m
    labels: { severity: warning }

  # 5. Duty-cycle breaches — GC, CPU starvation, lost core pinning.
  - alert: AeronConsensusCycleBreach
    expr: increase(cluster_cycle_time_threshold_exceeded_total[5m]) > 0
    labels: { severity: warning }

  - alert: AeronServiceCycleBreach
    expr: increase(cluster_service_cycle_time_threshold_exceeded_total[5m]) > 0
    labels: { severity: warning }

  - alert: AeronDriverCycleBreach
    expr: increase(driver_cycle_time_threshold_exceeded_total[5m]) > 0
    labels: { severity: warning }
    annotations:
      summary: "Driver {{ $labels.agent }} duty cycle exceeded 100ms"

  #    Is your code or Raft the bottleneck? Ratio, not absolutes.
  - alert: AeronServiceCodeIsBottleneck
    expr: |
      cluster_service_max_cycle_time_seconds
        / on(instance, clusterId) cluster_max_cycle_time_seconds > 5
    for: 5m
    labels: { severity: warning }

  # 6. Errors — alert on the count, then read the text with `ClusterTool errors`.
  - alert: AeronErrors
    expr: |
      increase(cluster_errors_total[5m]) > 0
        or increase(cluster_service_errors_total[5m]) > 0
        or increase(driver_errors_total[5m]) > 0
        or increase(archive_errors_total[5m]) > 0
    labels: { severity: warning }

  # 7. Recovery risk: snapshot age against your RTO budget, and slow snapshots.
  - alert: AeronSnapshotTooOld
    expr: time() - cluster_last_snapshot_timestamp_seconds > 3600
    labels: { severity: warning }

  - alert: AeronSnapshotTooSlow
    expr: |
      increase(cluster_snapshot_duration_threshold_exceeded_total[30m]) > 0
        or increase(cluster_service_snapshot_duration_threshold_exceeded_total[30m]) > 0
    labels: { severity: info }

  # 8. Transport underneath. Rate-of-change, never absolute totals.
  - alert: AeronBackPressure
    expr: rate(driver_sender_bpe_total[1m]) > 0
    for: 2m
    labels: { severity: warning }

  - alert: AeronPacketLoss
    expr: |
      rate(driver_nak_messages_received_total[1m]) > 0
        or rate(driver_retransmits_sent_total[1m]) > 0
    for: 2m
    labels: { severity: warning }

  #    These are documented as "expected to be zero" — any delta is real.
  - alert: AeronShouldBeZeroCounters
    expr: |
      increase(driver_invalid_packets_total[5m]) > 0
        or increase(driver_short_sends_total[5m]) > 0
        or increase(driver_loss_gap_fills_total[5m]) > 0
        or increase(driver_flow_control_over_runs_total[5m]) > 0
        or increase(cluster_timed_out_clients_total[5m]) > 0
    labels: { severity: warning }

  # 9. Node dark. Both ages are refreshed ~1/s, so >5s means the process is wedged or gone.
  - alert: AeronNodeDark
    expr: cluster_activity_age_seconds > 5 or driver_heartbeat_age_seconds > 5
    for: 10s
    labels: { severity: critical }

Calibrate the windows, don’t paste them blind

The for: durations and rate windows above are sized off Aeron’s documented timers (10 s heartbeat, 1 s election, 100 ms duty cycle, 1 s snapshot) plus the ~1 s counter refresh — but the multipliers are operator judgment, not published thresholds. Two that will bite you:

• rate(...[1m]) == 0 needs at least two samples in the window. At a 30 s scrape interval a 1 m window is two points and one missed scrape makes it NaN, silently disarming the alert. Keep the scrape at 5–15 s for cluster metrics.
• AeronCommitStalled will fire during a legitimate election. That is why it’s gated on role == 2 and state == 1 and given a for: longer than the election timeout. Loosen the for: before you loosen the gates.

Per-stream monitoring: the cluster’s own stream ids

Cluster traffic rides ordinary Aeron pub/sub, so the driver-level counters (snd-bpe, NAKs, retransmits, publisher/subscriber positions) are all available per stream — and the stream that is losing packets tells you which failure you have. These are the defaults; all are overridable via system property, so read them off your own config rather than assuming.

Stream id	Stream	Configured by	Loss here means
100	Cluster log (leader → followers)	aeron.cluster.log.stream.id	Replication is dragging — widening append−commit gap, then quorum loss
101	Ingress (clients → leader)	aeron.cluster.ingress.stream.id	Clients can’t reach the leader; back-pressure here throttles your gateway
102	Egress (leader → clients)	aeron.cluster.egress.stream.id	Acks/responses dropping — clients retry or time out
103	Service replay (archive → service)	aeron.cluster.replay.stream.id	Slow recovery/replay on restart
104	Service → consensus module	aeron.cluster.service.stream.id	Service can’t report progress to the CM
105	Consensus module → service	aeron.cluster.consensus.module.stream.id	Service isn’t being driven
106	Service snapshot	aeron.cluster.snapshot.stream.id	Snapshot write path
107	Consensus-module snapshot	(same property, different default)	Snapshot write path
108	Consensus (inter-node: votes, heartbeats, canvass)	aeron.cluster.consensus.stream.id	Loss here is what actually triggers elections — heartbeats and vote requests are dropping

The 106/107 split is a defaults-only artifact: the consensus module and the service container both read the same aeron.cluster.snapshot.stream.id property, they just default to different values (107 and 106). Set that property explicitly and both collapse onto one stream id — so a dashboard that keys the two snapshot paths apart by stream id breaks the moment someone configures it.

Key Insight: 108 vs 100 tells you the shape of the incident

Loss on the log stream (100) is a throughput problem: followers fall behind, the append−commit gap grows, back-pressure builds, but the cluster stays up. Loss on the consensus stream (108) is an availability problem: dropped heartbeats and vote messages are how a healthy cluster ends up in an election loop. Same network, same NAK counters — completely different incident. Split your loss panel by stream id (driver_sender_bpe_total{streamId} is exported per-stream) and this distinction is free.

Correlate with the OS, or you’ll misdiagnose

Every “the cluster is unstable” incident whose real cause is GC, a stolen core, or a saturated NIC looks identical in Aeron’s counters: cycle-time breaches and elections. The counters tell you the duty cycle stalled; they cannot tell you why. Aeron’s own troubleshooting guide points at run-queue length, swap, and thread scheduling for exactly this reason — so scrape these alongside, on the same timeline:

Check	Command	What you’re looking for
Run queue / context switches	vmstat 1	r column persistently above your core count ⇒ the driver and CM threads are being descheduled. cs spiking ⇒ something else landed on their cores. Any non-zero si/so (swap) is disqualifying for a low-latency node.
Per-core steal / softirq	mpstat -P ALL 1	%steal on a virtualised host, or %soft/%irq on the cores you isolated — either means you don’t actually own the core you pinned to.
UDP datagram errors	sar -n UDP 1 / netstat -su	idgm_err, receive-buffer errors, and noport climbing in step with Aeron’s NAKs confirms the loss is below Aeron — kernel socket buffer or NIC ring, not the protocol. See receive-path buffers.
NIC drops	ethtool -S <if> | grep -i drop, ip -s link	rx_dropped/rx_no_buffer ⇒ ring buffer too small or the receive path isn’t keeping up.
GC pauses	JVM GC log (-Xlog:gc*, or -Xlog:safepoint)	Any pause approaching the 100 ms duty-cycle threshold explains a cycle breach; anything near the 10 s heartbeat timeout explains an election. Safepoint time is the number that matters, not GC time.
Disk latency (followers too)	iostat -x 1	await on the archive volume — a slow follower disk caps the whole cluster, because commit waits on a quorum and the quorum waits on the slowest acker.

Key Insight: the counters are the symptom, the OS is usually the cause

Aeron’s cluster counters are extremely good at telling you that the duty cycle stalled and when. They are structurally incapable of telling you it was a 180 ms safepoint pause, or that isolcpus silently stopped applying after a kernel upgrade. If your on-call dashboard shows Aeron metrics without run queue, steal time, safepoint pauses, and UDP errors on the same time axis, every incident starts with a 20-minute detour. Put them on one board.

What to actually put on the dashboard

If you build one panel set, make it this — ordered top to bottom the way you’d triage:

1. Is it up? Node role per node (exactly one Leader), liveness/heartbeat age, scrape targets up.
2. Is it progressing? Commit-position rate (your real throughput) and the append−commit gap.
3. Is it stable? Election count (flat line = healthy) and consensus-module state.
4. Is your code the bottleneck? Service cycle time vs consensus cycle time — if service ≫ consensus, the problem is in your handler, not Raft.
5. Is the substrate healthy? Back-pressure, NAK/retransmit rate, archive/disk free space — split by stream id, so consensus-stream loss (elections) is visibly distinct from log-stream loss (lag).
6. Can you recover fast? Snapshot age, valid-snapshot count, snapshot-duration breaches.
7. Is the machine to blame? Run queue, %steal, safepoint pauses, UDP errors — on the same time axis as everything above, or you’ll spend the first twenty minutes of each incident finding out.

Key Insight: monitor before you tune

These same counters are your tuning feedback loop. Change one knob, watch the relevant counter, keep the change only if the signal improves — cycle times and retransmits move p99; back-pressure and stream positions move p50/throughput. See Performance Tuning. The dashboard you build for incidents is the dashboard you tune against.

Sources

• Counter model: CncFileDescriptor, Agrona CountersReader, AeronStat, and the Monitoring and Debugging wiki (“read by any external process with no significant impact on performance”).
• Counter names/type ids: AeronCounters.java, SystemCounterDescriptor.java, ConsensusModule.java (labels + defaults: 10 s heartbeat, 100 ms cycle, 1 s snapshot), and Understanding Cluster Counters / The Aeron Files — Counters (append position = Archive rec-pos).
• DIY exporter guidance: Aeron cookbook — Read counters (names Datadog + Prometheus). Community exporters: singleplayer88/aeron-exporter, fairtide/aeron-prometheus-stat, scalecube/scalecube-metrics.
• Alert bases: Cluster errors (“No catchup progress”; storage-exhaustion → ClusterTerminationException), Cluster troubleshooting.
• Stream ids, state enum, and counter type ids in this page were read off ConsensusModule.java (State 0–6; log 100, consensus 108, snapshot 107; aeron.cluster.cycle.threshold = 100 ms), AeronCluster.java (ingress 101, egress 102), ClusteredServiceContainer.java (replay 103, service 104, consensus-module 105, snapshot 106), AeronCounters.java (213 client-timeout, 236/237 per-service snapshot duration, 239 leadership-term id), and SystemCounterDescriptor.java (Loss gap fills, Invalid packets, conductor/sender/receiver cycle-time labels; aeron.driver.*.cycle.threshold = 100 ms).
• Premium: Aeron Insights overview (beta; three components), Prometheus exporters (reads CnC file; HTTP-server vs push-client), Aeron metrics (verbatim cluster_* / driver_* metric names and descriptions).

Members only

The rest of this page is for approved members.

Still on this page:

1. How Aeron exposes telemetry: the CnC file
2. Path 1 — OSS: write (or reuse) a counter exporter
3. Path 2 — Premium: Aeron Insights
4. The counters that matter (and what bad looks like)
5. Alerting: mapping metrics to failure modes
6. Alert rules you can paste
7. Per-stream monitoring: the cluster’s own stream ids
8. Correlate with the OS, or you’ll misdiagnose
9. What to actually put on the dashboard
10. Sources

Log in or request access

Sign in with an emailed link, no password. New accounts are approved by hand.