Snapshot Failure and Recovery in Aeron® Cluster

A node dies in the middle of taking a snapshot — kill -9, OOM-killer, power loss, or an exception thrown from your own onTakeSnapshot. What state is it in, and what do you have to do to bring it back? The short answer is reassuring: an interrupted snapshot is designed to be invisible. The node restarts from the previous valid snapshot plus log replay, with no manual intervention. The longer answer — what’s left behind, when you do need ClusterTool, and what changes with Aeron Premium — is this page.

Note

This page is the operational runbook. For protocol-level snapshot mechanics (recordings, the recording log, replay), see The Aeron Files — we link, we don’t duplicate. For silent snapshot corruption caused by application bugs, see the state-and-determinism runbook. Behaviour described here was verified against Aeron OSS 1.51.1 (June 2026).

Why an interrupted snapshot is safe by design

Two design facts do all the work. Everything in the runbook below follows from them.

1. A snapshot is a set of local archive recordings. The SNAPSHOT action is appended to the Raft log by the leader, so every node snapshots at the same log position — but each node writes its snapshot to its own local archive, independently. Your service serializes state in onTakeSnapshot to a recorded publication; the consensus module records its own state (sessions, timers) separately.

2. The recording.log entry is written only after everything completes. A snapshot only “exists” once entries for it are appended to the node’s recording.log file — and the consensus module appends them after all service recordings and its own recording are fully written, with its own entry written last (the file is fsynced only when aeron.cluster.file.sync.level > 0 — the default is 0). Recovery builds its plan exclusively from complete, valid entry sets. As Aeron maintainer Dmytro Vyazelenko puts it: “Snapshot is considered complete only if all services complete their snapshot without an error followed by a successful snapshot of the consensus module state. Partial snapshots are ignored when creating a recovery plan.”

Kill the process at any point before that final append and the outcome is identical: no entry, no snapshot, and on restart the recovery plan simply uses the previous valid snapshot (or full log replay if none exists). There is no “half snapshot” state to repair.

Key Insight

The recording log is an append-only commit protocol for snapshots. The consensus-module entry acts as the commit marker — it is written last, so a torn write can never produce a valid-looking partial snapshot. This is why the default recovery answer is “just restart the node.”

Failure modes during a snapshot

What actually happens depends on how the snapshot dies. The consensus module distinguishes terminal from non-terminal errors — and the direction of the failure (taking vs. loading) matters a lot.

Scenario	Node behaviour	Snapshot outcome	Operator action
Process killed mid-snapshot (kill -9, OOM, power loss)	Node down	No recording.log entry — interrupted snapshot invisible	Restart the node (Case A)
Exception in onTakeSnapshot (serialization bug, etc.)	Node keeps running; error logged	Service acks NULL → consensus module aborts the set, no entry	Fix the bug, retake snapshot (Case B)
Disk full during snapshot (ArchiveException: STORAGE_SPACE)	Terminal — agent terminates, node exits	No entry	Free/expand disk, restart
Archive won’t start after unclean stop (straddling page boundary)	Crash-loop at startup	Torn last write (usually power loss, not SIGKILL)	ArchiveTool verify + compact (Case A)
Exception while loading a snapshot at startup	Terminal — AgentTerminationException, node exits (crash-loop if retried)	Latest entry exists but is unloadable	invalidate-latest-snapshot (Case C)
Local cluster/archive data lost or unrecoverable	Node down	—	Wipe and replay, or rebuild from backup (Case D)

Two properties worth calling out, both consequences of snapshots being per-node local actions:

• One node’s failure doesn’t stop the others. Service acks flow over local IPC between a node’s own container and its own consensus module; nothing in the snapshot path propagates failure to other members. A follower that fails its snapshot leaves the leader and the other follower with perfectly good snapshot entries — the cluster keeps serving traffic throughout, since only quorum matters.
• Nodes can legitimately disagree about which snapshots exist. If the leader’s snapshot failed but the followers’ succeeded, each node recovers from its own recording log. That’s normal, not divergence — all paths converge to the same state via log replay.

Does it matter which role is killed?

For the recovery procedure — no. Snapshots, recordings, and the recording log are per-node local, so a killed node recovers the same way whatever its former role: restart → previous valid snapshot + log replay. The SNAPSHOT action goes through the log, so every node is snapshotting at the same position at once — “killed during snapshot” is never leader-specific in the first place.

Role only changes cluster availability during the kill, not the killed node’s recovery. Kill the leader and the cluster runs a brief election; the node rejoins as a follower afterward. Kill a follower and quorum is untouched (2 of 3), so there isn’t even a blip. Either way the killed node’s local recovery is identical. (Watch out for the trap where a bug looks leader-specific — Aeron maintainers have closed “messages dropped after killing the leader” reports as non-deterministic service code: your ClusteredService must behave identically regardless of role.)

Take vs. load — opposite severities

A failure while taking a snapshot is benign: the node logs the error and keeps processing the log — only storage exhaustion or an explicit AgentTerminationException is terminal. (Service-side exceptions have long been non-fatal; exceptions in the consensus module’s own snapshot became non-terminal in 1.49.0.) A failure while loading a snapshot is always terminal — the node cannot start with half-restored state. If a node crash-loops at startup, you are in Case C, not Case A.

The recovery runbook

First, diagnose. The first two read the node’s cluster directory as plain files and work while the node is fully down; the last two connect to the local archive, so they need the node’s media driver and archive running (fine when only the cluster services crashed, not after a whole-process kill):

CLUSTER_DIR=/srv/node0/cluster
CP=aeron-all-1.51.1.jar

java -cp $CP io.aeron.cluster.ClusterTool $CLUSTER_DIR errors          # why did it die? (offline)
java -cp $CP io.aeron.cluster.ClusterTool $CLUSTER_DIR recording-log   # what snapshots exist? (type=1 entries, isValid) (offline)
java -cp $CP io.aeron.cluster.ClusterTool $CLUSTER_DIR recovery-plan 1 # what would recovery use? (arg = service count; needs archive up)
java -cp $CP io.aeron.cluster.ClusterTool $CLUSTER_DIR describe-latest-cm-snapshot  # replay & print the CM snapshot (needs archive up)

Then pick the case that matches.

Case A: node killed mid-snapshot — restart it

Procedure: restart the process. That’s it. On startup the node builds a recovery plan from its recording log (the interrupted snapshot isn’t in it), loads the latest valid snapshot, replays its local log recording from the snapshot position, then joins the live log — pulling any terms it missed from the leader’s archive and catching up. You’ll see the node re-executing application messages in its logs; that is the replay doing its job.

The only cost is time: recovery length is set by snapshot age (replay distance) plus catch-up. Nothing about the interrupted snapshot needs cleaning before restart.

Why restart works even though the archive was killed mid-write: the interrupted snapshot recording is left “open” (stopPosition = -1) in the archive catalog, and on the next start the archive scans the last segment, computes the real stop position, and repairs the descriptor automatically. As an Aeron contributor put it, “in order to ‘close’ a recording, we just need to start recording again” — the archive notices the previous writer is gone and closes it for you. No manual catalog surgery.

Optional housekeeping: the interrupted attempt leaves orphaned recordings in the local archive (the archive repairs their stop positions on restart but never deletes them). They waste disk, nothing more. To reclaim: identify recordings on the snapshot channel (alias=snapshot) whose IDs appear in ArchiveTool describe but in no recording-log entry, purge them via AeronArchive.purgeRecording, then ArchiveTool compact. This identification procedure is our synthesis — Aeron ships no official “failed-snapshot cleanup” guide.

When a restart isn’t enough: the archive won’t start

Occasionally the archive itself refuses to start and the node crash-loops with an ArchiveException. This is the real “kill-9 mid-snapshot broke my node” story people report — but it’s the exception, not the rule, and it splits into two very different causes:

• Torn last write — ArchiveException: Found potentially incomplete last fragment straddling page boundary in file: …, with a runtime hint Run 'ArchiveTool verify' for corrective action!. The auto-repair above can’t tell a zero-filled tail from a half-written frame, so it refuses to guess. This is overwhelmingly a power-loss / VM-crash / networked-volume hazard, not plain SIGKILL: with the default aeron.archive.file.sync.level=0, writes live in the OS page cache, which a kill -9 leaves intact for the kernel to flush — only real power loss loses them. Fix: run ArchiveTool verify (marks faulty entries INVALID) then ArchiveTool compact, and restart. Prevent: set aeron.archive.record.checksum=CRC-32C, and raise aeron.archive.file.sync.level and aeron.archive.catalog.file.sync.level to 1–2 if you must survive power loss (at a write-path latency cost — see host & OS parameters).
• Replay-position mismatch — ArchiveException: requested replay start position=… is less than recording start position=…. This does not come from a single interrupted snapshot; it appears when the log was truncated/purged and a node then needs to replay from a position that no longer exists. That’s Case D territory, not Case A — see below.

The “must copy from a good node” claim is for a different failure

A widely-circulated recovery note tells you to rsync a healthy node’s archive/+cluster/ directories onto the broken one. That advice is for the multi-node state-loss scenario (files lost on some nodes and the log truncated on the rest — aeron#1533), which is genuinely “not supported, restore from backup.” It is not the fix for a single node killed mid-snapshot — that node recovers by simply restarting. Don’t reach for directory-copying (which is also fraught, since each node’s state carries its own identity/mark file) until you’ve confirmed a plain restart fails and you’re actually in the truncated-log case.

Case B: exception in onTakeSnapshot — node still running

The node is still up and processing; what failed is the snapshot attempt itself. The consensus module logged service=N failed to take snapshot (visible in ClusterTool errors) and appended nothing to the recording log.

1. Check errors on the node to get the actual exception from your service code.
2. Fix the bug. A throwing onTakeSnapshot is almost always an application serialization defect — and the same class of bug that causes silent snapshot divergence when it doesn’t throw.
3. Retake the snapshot (ClusterTool <leader-cluster-dir> snapshot) once deployed. Until then the cluster runs fine but its recovery replay window keeps growing — an old snapshot means a long restart.

Case C: completed snapshot is corrupt — invalidate it

This is the case where a snapshot did commit (entries exist, isValid=true) but the node dies loading it — snapshot ended unexpectedly, a deserialization exception in onStart, a crash-loop. The snapshot is bad even though the protocol completed; recovery must be pointed at the previous one.

1. Stop the node. invalidate-latest-snapshot writes the recording.log file directly; it does not coordinate with a running consensus module.
2. Invalidate:

   java -cp $CP io.aeron.cluster.ClusterTool $CLUSTER_DIR invalidate-latest-snapshot
   # → invalidate latest snapshot: true

   This flips the invalid bit on the latest snapshot’s consensus-module entry and all its service entries, so recovery falls back to the prior snapshot — or full log replay if none remains. Run it again to step back two snapshots.
3. Restart and budget for a longer replay.

If the corruption is logical (a serialization bug baked into the snapshot content), the same bad snapshot exists on every node — stop all nodes, invalidate on each node’s cluster directory, then restart them together. That all-nodes-while-stopped procedure is exactly what Aeron’s own system tests do; the official docs don’t spell it out, so treat the tests as the reference.

Invalidation is per-node file surgery

The tool edits one node’s recording.log. Invalidate on one node and the others will still happily load the snapshot you distrust. Decide first whether the snapshot is bad on this disk (one node) or bad everywhere (all nodes), and match the blast radius.

Case D: local data unrecoverable — wipe or rebuild

When the archive or cluster directory itself is damaged beyond the snapshot chain, OSS gives you two paths, and the difference is whether the full log still exists on the leader:

• Full log retained → wipe the node’s cluster and archive directories and start it empty. The node rejoins and the leader replays the log to it from position 0 (an Aeron system test verifies exactly this). Caveats: OSS removed dynamic membership in 2023, so the replacement must reuse the dead member’s endpoints; snapshots are not replicated to an empty OSS node, so the full-history replay must be deterministic under your current code version; and replay-from-zero can take hours on a mature system.
• Log truncated/purged to a snapshot → an empty node cannot rejoin; the replay it needs no longer exists. Rebuild from Cluster Backup (trigger a fresh snapshot, let the backup node replicate it, copy its archive + recording.log onto the replacement) or copy the data directories from a healthy stopped peer. Martin Thompson’s guidance: a node that loses storage “needs to have the leader replay the log to it from the beginning of time” — otherwise you need a backup, and “ClusterBackup provides a basic mechanism to do this.”

This is also why seed-recording-log-from-snapshot — the tool that rebases a node’s recording log onto its latest snapshot so old log recordings can be purged — must be used with care: purge the log on the leader and you have removed Case D’s first path for every future rebuild.

OSS vs. Premium: same failure semantics, different operational options

Does Aeron Premium handle a killed-mid-snapshot node differently? No. Premium is built on top of the open-source cluster core; the snapshot commit protocol, the recording-log invalidation model, and the terminal/non-terminal error split above are the same code path. Nothing in the public Premium documentation describes different snapshot-failure handling — and the snapshot-integrity tooling (invalidate-latest-snapshot, validate-recording-log, describe-latest-cm-snapshot) is all OSS. (Premium internals sit behind the subscription, so we can only attest to what’s publicly documented — but everything public points at the same core.)

What Premium changes is everything around the failure — how often you snapshot, where a good snapshot can come from, and how fast you can rebuild:

	OSS	Premium (Cluster Standby et al.)
Interrupted-snapshot semantics	No recording-log entry; restart recovers from previous snapshot	Identical — same consensus-module core
Who takes snapshots	Every node, including the leader — a cluster-wide pause	Standby snapshots: taken on non-voting standby nodes; the cluster never pauses
Snapshot sources at recovery	Local archive only	Local, plus a restarting node detects newer standby snapshots and replicates one from the standby before loading — or pre-fetch with replicate-standby-snapshot
Stale/invalidated snapshot chain	Fall back to older snapshot + longer replay	A newer standby snapshot can paper over a bad local chain; invalidate-latest-snapshot also invalidates standby snapshot entries (since 1.47.0)
Snapshot retention vs. log purging	Manual discipline (seed-recording-log-from-snapshot + archive purge)	Data Retention Regulator policy keeps the last N valid snapshots plus the log needed for recovery
Snapshot health monitoring	ClusterTool errors, counters (see Monitoring & Alerting)	Aeron Insights (beta) Prometheus metrics — valid-snapshot count, last-snapshot timestamp, snapshot-duration threshold breaches
Rebuilding a dead member	ClusterBackup + manual data copy, static endpoints	Standby node converts in place (transition-as-member) — no replay-from-zero

Key Insight

Premium doesn’t make snapshot failure safer — OSS is already safe by construction. It makes snapshot failure cheaper: standby snapshots mean you can snapshot far more often without touching hot-path p99, and a fleet that snapshots hourly instead of daily has an hourly-sized replay window when a node dies mid-snapshot. The frequency you can afford, not the failure handling, is the real OSS/Premium gap.

How this moves your numbers

• Recovery time (availability) — An interrupted snapshot costs nothing at failure time; the bill arrives as replay length on restart, which is set by the age of the last valid snapshot. Each invalidation step (Case C) doubles down on that: one snapshot back, one replay window longer.
• p99 / max latency — In OSS, retaking a failed snapshot means another stop-the-world pause on the hot path; on a busy matching engine that pause lands squarely on max latency. This — not failure semantics — is what pushes production trading systems toward standby snapshots.
• Throughput — Unaffected by snapshot failure itself; the cluster keeps committing while a follower’s snapshot attempt dies. Watch instead for the disk-full terminal case, which takes the node (and, on the leader, triggers an election) with it.

Sources

• ConsensusModuleAgent.java (snapshot-on-ack protocol, recording-log append ordering, terminal vs. non-terminal error handling) and ClusteredServiceAgent.java (onTakeSnapshot exception path, NULL service ack), Aeron OSS master at 1.51.1.
• ClusterTool.java and RecordingLog.java (command semantics, invalidateLatestSnapshot, recovery-plan validation).
• Maintainer statements: aeron#1942 (partial snapshots ignored; non-terminal error behaviour), aeron#1263 (snapshot trigger via the log; per-node archive recordings), aeron#1533 (multi-node state-loss “not supported” — the “copy from a good node” scenario), and Martin Thompson on storage loss (Stack Overflow).
• Archive won’t start after unclean stop: aeron#1812 (Vyazelenko on the “straddling page boundary” root cause + record.checksum=CRC-32C fix), aeron#1657 (replay-position mismatch loop), killed-recording auto-close (Stack Overflow), and file.sync.level=0 power-loss risk (Stack Overflow, Martin Thompson). Catalog.refreshAndFixDescriptor / Archive.java (FILE_SYNC_LEVEL_DEFAULT = 0) and ArchiveTool (verify, compact, mark-invalid) in the OSS source.
• Aeron Cluster Tutorial (§ Snapshots, recording-log entries) and Operating Aeron Cluster (ClusterTool usage), Cluster Backup (node re-provisioning), Cluster system tests (invalidate-on-all-nodes reference procedure).
• Premium: Cluster Standby overview (standby snapshots, replication on restart, transition-as-member), Data Retention Regulator, Aeron Insights metrics, Aeron CHANGELOG (1.49.0 non-terminal snapshot exceptions; 1.47.0 standby-snapshot invalidation).
• Background: The Aeron Files — Snapshots.