Monitoring Aeron®: Metrics and Alerting for Your Observability Platform

Aeron exposes a rich set of lock-free counters in shared memory — but it ships no /metrics endpoint. To get those signals onto your company’s Prometheus/Grafana/Datadog stack you either write a small exporter (OSS) or run Aeron Insights (Premium). This page covers both paths, then maps the counters that actually matter to the failure modes they predict and the alert conditions to set.

Note

This is the platform-integration and alerting page. For reading counters interactively at the console, the drafted aeron-stat and Cluster Diagnostic Counters pages go deeper on individual signals; for the protocol-level meaning of each counter, The Aeron Files has a dedicated section. Verified against Aeron OSS 1.51.1 and Aeron Insights 0.3.0-beta (2026).

How Aeron exposes telemetry: the CnC file

Every Aeron media driver maintains a command-and-control file, cnc.dat, memory-mapped in its Aeron directory. Among its regions are a counters metadata buffer, a counters values buffer, and an error-log buffer. Counters are plain 64-bit values updated lock-free by the driver; any external process can memory-map the file read-only and read them live, with — in Aeron’s own words — “no significant impact on performance.”

That is the whole integration surface. There is no push, no agent protocol, no HTTP — you map the file and iterate. The reference reader is AeronStat (the aeron-stat script), which loops once per second and prints every counter. Under the hood it does exactly what any exporter does:

// The canonical read pattern — Agrona's CountersReader over the mapped cnc.dat
final CountersReader counters = cncFileReader.countersReader();
counters.forEach((counterId, typeId, keyBuffer, label) -> {
    final long value = counters.getCounterValue(counterId);
    // → emit {counterId, typeId, label, value} to your metrics client
});

Counters are grouped by type id: 0–99 client/driver, 100–199 archive, 200–299 cluster. That type id is your routing key when you turn counters into metrics.

Key Insight

The CnC file is the API. Because reads are lock-free map-and-poll against shared memory, a scraper adds no back-pressure and no latency to the driver — you can poll it aggressively without touching your p99. This is why the DIY-exporter pattern below is cheap and safe, not a compromise.

Path 1 — OSS: write (or reuse) a counter exporter

Aeron OSS has no native Prometheus exporter. This is by design, and Aeron’s own cookbook points you at the DIY route: “Read the counter data using CountersReader” and feed a metrics system “such as DataDog or Prometheus.” The pattern is a per-node sidecar that maps that node’s cnc.dat, iterates counters on a timer, and exposes /metrics (Prometheus scrape) or pushes to StatsD/Datadog/CloudWatch.

You don’t have to write it from scratch — a couple of community exporters already implement this:

Exporter	What it is	State
singleplayer88/aeron-exporter	Reads cnc.dat via CncFileReader/CountersReader, serves /metrics (Jetty). Apache-2.0.	Recently maintained (2026)
fairtide/aeron-prometheus-stat	Same CnC-reader pattern, renames counters to Prometheus conventions; ships Docker Compose. Apache-2.0.	Older (~2023)
scalecube/scalecube-metrics	Telemetry library over Agrona counters + HdrHistogram, with a Prometheus-over-Aeron demo.	Actively maintained

Vet before you trust

These are community projects, not Aeron releases — pin a commit, read the code, and confirm it handles CnC version-major mismatches (the file format is versioned; a reader built against a different major throws on map). Treat any third-party exporter as sample code you own, not a supported dependency.

For Datadog, StatsD, or CloudWatch there is no off-the-shelf integration — the same CountersReader.forEach loop pushes to that platform’s client instead of exposing /metrics. A log-scraping fallback (parse the once-per-second aeron-stat console output) works but is strictly worse: coarser, lossy, and it re-parses text you already have as numbers.

Path 2 — Premium: Aeron Insights

Aeron Insights (Premium, currently beta) is the turnkey version of Path 1. It ships Prometheus exporters, a sample Grafana dashboard, and a CLI. The exporters “gather data by reading the files created by Aeron, principally the CnC file” plus the archive/cluster mark files — i.e. the same read-only approach, packaged and supported — in two per-node delivery models:

• HTTP-server exporter (default :8080, localhost) — Prometheus scrapes each node directly. This is the common approach.
• Push-client exporter — pushes to a Prometheus Push Gateway on an interval (default 1s).

The payoff is that Insights emits named, documented Prometheus metrics out of the box (cluster_*, driver_*, archive_*), with a Grafana “Cluster Overview” dashboard, instead of raw type-id’d counters you have to name and curate yourself.

	OSS DIY exporter	Premium Aeron Insights
Reads	cnc.dat (via your code)	cnc.dat + mark files (packaged)
Metric names	You define them	Documented cluster_* / driver_* / archive_*
Grafana dashboard	You build it	Sample “Cluster Overview” ships
Delivery	Whatever you write	HTTP scrape or Push Gateway
Support	Community / self	Adaptive (beta)
Cost	Free (Apache-2.0)	Subscription

Key Insight

The telemetry source is identical — both read the same CnC counters. What Premium sells here is the last mile: names, dashboards, packaging, and support. If you already run a Prometheus stack and have the engineering to own ~200 lines of exporter, OSS gets you the same signals. Insights is worth it when you want a supported, dashboard-ready path and don’t want to curate metric names across upgrades.

The counters that matter (and what bad looks like)

You do not need all of Aeron’s counters on a dashboard. For a clustered matching engine, these are the load-bearing ones. OSS names are the counter labels / type ids from the CnC file; the Insights column is the Prometheus metric name where one exists.

Signal	OSS counter (type id)	Insights metric	Read it as
Commit position	Cluster commit-pos: (203)	cluster_commit_position_total	The heartbeat of the cluster — quorum-replicated log position
Append position	Archive rec-pos of the log recording	(no dedicated metric)	Where the leader has appended; gap vs commit = follower lag
Node role	Cluster node role (201)	cluster_node_role	2 = Leader, 0 = Follower, 1 = Candidate
Consensus state	Consensus Module state (200)	cluster_consensus_module_state	1 = Active, 3 = taking a snapshot
Election state / count	Cluster election state (207) / election count (238)	cluster_election_state	Steady state = 17 (Closed); count should never climb
CM cycle time	Cluster max cycle time in ns (216) / exceeded (217)	cluster_max_cycle_time_seconds	Duty-cycle latency of the consensus module
Service cycle time	Cluster container max cycle time in ns (218) / exceeded (219)	(service-labelled cycle metric)	Duty-cycle latency of your service code
Errors	Cluster Errors (212) / Cluster Container Errors (215)	cluster_errors_total	Monotonic tally — any delta warrants a look
Valid snapshots	Cluster snapshot count (205)	cluster_valid_snapshots	Count of usable snapshots
Snapshot age	(derive from recording-log timestamp)	cluster_last_snapshot_timestamp_seconds	Old snapshot ⇒ long replay on recovery
Snapshot duration	Total max snapshot duration in ns (234) / exceeded (235)	cluster_snapshot_duration_threshold_exceeded_total	Long snapshots stall the hot path (OSS)
Back-pressure	snd-bpe (13) / Sender flow control limits	driver_sender_bpe_total	”Should be zero in a healthy system”
Loss / NAK	NAKs sent/received, Retransmits sent (type 0)	driver_nak_messages_*_total, driver_retransmits_sent_total	Network loss ⇒ replication drag
Liveness	(freshness of counter updates)	cluster_activity_age_seconds, driver_heartbeat_age_seconds	Seconds since the component was last seen active

Two things people get wrong

“Appended position” is not a cluster counter — it’s the Aeron Archive rec-pos counter for the log recording. Compute the follower-lag gap as rec-pos(log) − commit-pos; neither OSS nor Insights gives you the gap pre-computed. And the error text lives in the distinct-error log (dumped point-in-time by ClusterTool errors), not in a metric — the Cluster Errors counter is only a count. Alert on the count; page a human to read the text.

Alerting: mapping metrics to failure modes

Each of these ties a metric to a failure-mode runbook scenario. Aeron publishes almost no numeric thresholds — the timers below are documented; the rest are operator judgment, so calibrate against your own baseline and start loose to avoid alert fatigue.

Alert	Condition	Why / documented basis
Commit stall (Critical)	commit-pos delta == 0 while state == Active for > ~10 s	Aeron’s own “No catchup progress” warning fires when commit doesn’t advance for longer than the leader heartbeat timeout (default 10 s). Causes: slow app code, machine load, GC.
No leader (Critical)	No node reports role == Leader (2) for > election timeout	Election in progress = cluster not committing.
Election churn (High)	election count (238) increasing, or election state not returning to 17 (Closed)	Any increment is a p99 cliff. Triggers: heartbeat timeout, lost quorum. (“Election storm” is not a documented term — set your own rate.)
Follower lag (Medium)	rec-pos(log) − commit-pos growing over time	Leader appending faster than a quorum acks — heading toward back-pressure or quorum loss. No documented gap threshold.
Cycle-time breach (Medium)	cycle time exceeded counters (217/219) incrementing	Duty-cycle threshold default is 100 ms for CM and service. Spikes = GC / CPU starvation; a 10 ms cycle is a 10 ms hole in p99.
Errors rising (High)	Cluster Errors (212) delta > 0	Then read the text via ClusterTool errors. No numeric rate documented.
Snapshot too old (Medium)	now − cluster_last_snapshot_timestamp_seconds > your recovery budget	Snapshots are manual in Aeron — an old snapshot means a long log replay on restart. See Snapshot Failure and Recovery.
Snapshot too slow (Low/Medium)	snapshot-duration exceeded (235) incrementing	Threshold default 1 s. In OSS the snapshot stalls the hot path; on the leader that lands on max latency.
Back-pressure / loss (Medium)	snd-bpe, NAK, or retransmit rate rising above baseline	Subscribers/followers not keeping up, or network loss dragging replication. No documented rate threshold.
Storage exhaustion (Critical)	Free space on archive/cluster volumes below your floor	Archive storage exhaustion throws ClusterTerminationException and terminates the node — the one snapshot/archive failure that is always terminal. Alert before it happens.
Node dark (Critical)	cluster_activity_age_seconds / driver_heartbeat_age_seconds climbing, or scrape target down	The component stopped updating counters — process dead or wedged.

The four timers are different — don’t conflate them

Leader heartbeat timeout = 10 s. Election timeout = 1 s. Duty-cycle threshold = 100 ms. Snapshot-duration threshold = 1 s. These are separate knobs at separate layers; an alert that confuses them (e.g. treating a 100 ms cycle breach as a 10 s stall) will either page constantly or miss the real outage. Insights’ processing-time metrics default to a 1 s threshold, distinct from the OSS 100 ms duty-cycle default — be explicit about which layer a threshold belongs to.

What to actually put on the dashboard

If you build one panel set, make it this — ordered top to bottom the way you’d triage:

1. Is it up? Node role per node (exactly one Leader), liveness/heartbeat age, scrape targets up.
2. Is it progressing? Commit-position rate (your real throughput) and the append−commit gap.
3. Is it stable? Election count (flat line = healthy) and consensus-module state.
4. Is your code the bottleneck? Service cycle time vs consensus cycle time — if service ≫ consensus, the problem is in your handler, not Raft.
5. Is the substrate healthy? Back-pressure, NAK/retransmit rate, archive/disk free space.
6. Can you recover fast? Snapshot age, valid-snapshot count, snapshot-duration breaches.

Key Insight: monitor before you tune

These same counters are your tuning feedback loop. Change one knob, watch the relevant counter, keep the change only if the signal improves — cycle times and retransmits move p99; back-pressure and stream positions move p50/throughput. See Performance Tuning. The dashboard you build for incidents is the dashboard you tune against.

Sources

• Counter model: CncFileDescriptor, Agrona CountersReader, AeronStat, and the Monitoring and Debugging wiki (“read by any external process with no significant impact on performance”).
• Counter names/type ids: AeronCounters.java, SystemCounterDescriptor.java, ConsensusModule.java (labels + defaults: 10 s heartbeat, 100 ms cycle, 1 s snapshot), and Understanding Cluster Counters / The Aeron Files — Counters (append position = Archive rec-pos).
• DIY exporter guidance: Aeron cookbook — Read counters (names Datadog + Prometheus). Community exporters: singleplayer88/aeron-exporter, fairtide/aeron-prometheus-stat, scalecube/scalecube-metrics.
• Alert bases: Cluster errors (“No catchup progress”; storage-exhaustion → ClusterTerminationException), Cluster troubleshooting.
• Premium: Aeron Insights overview (beta; three components), Prometheus exporters (reads CnC file; HTTP-server vs push-client), Aeron metrics (verbatim cluster_* / driver_* metric names and descriptions).