Archive Retention and Purge: How Much Log Can You Delete?

Your archive volume is filling up and someone wants to delete old cluster-log segments. The archive API will happily let you. purgeSegments knows about segment boundaries and active recordings — it knows nothing about whether the bytes you are deleting are the only thing standing between a node and a rebuild.

This page is the missing check. It gives you the positions to measure, the invariant that decides restart-vs-rebuild, and a purge procedure that fails safe. For what happens after a node fails, see Snapshot Failure and Recovery; for the backup that underwrites the rebuild path, Backups and Disaster Recovery.

Note

Verified against Aeron OSS master (1.51.x). Archive and cluster internals — how recordings, segments, and the recording log are laid out — are covered by The Aeron Files; this page is the operational arithmetic on top.

The one question this scenario turns on

Before you delete anything, and before you start a node that has been offline, you are answering a single question: can this node restart normally, or must it be rebuilt?

Node’s local state	Live source retains log from ≤ its position?	Verdict
Complete snapshot set + contiguous local log	Yes (L ≤ P)	Normal restart. Local snapshot load → local replay → leader catch-up.
Complete snapshot set + contiguous local log	No (L > P)	Rebuild. Catch-up has a gap that no restart will close.
No snapshot, local log contiguous from position 0	Yes	Normal restart, full local replay. Slow but valid.
No snapshot, local log starts after 0	—	Rebuild unless this is a deliberately rebased, tested image.
Latest snapshot commits but won’t load	Prior set + its log retained	invalidate-latest-snapshot, then re-test this table.
Empty disk (replaced instance)	Live source retains from 0	Restart works, but usually slower than provisioning a backup.
Empty disk	Live source purged past 0	Rebuild from a verified image. No exceptions.

Everything below is how to measure L and P, and how to purge without ever pushing a node from row 1 into row 2.

Election catch-up transfers log, never the leader’s snapshot

This is the fact that makes the whole page necessary, and it surprises people. There is no election state in which a joining node asks the leader for its latest cluster snapshot. A node uses a snapshot only if that snapshot is reachable through its own recovery metadata and archive.

So a blank node does not get bootstrapped from a healthy peer’s snapshot. It gets rebuilt from log, and if the log it needs has been purged, it cannot join at all. Deleting log is therefore deleting other nodes’ ability to return — which is exactly the coupling purgeSegments cannot see.

(Premium standby snapshots are the one qualification: a restarting node can replicate a standby snapshot, but only when its local recording.log already describes that remote recording. A truly empty disk still needs provisioning. See Cluster Standby.)

The five positions

Retention decisions made on age — “keep seven days” — are proxies for the thing that actually matters, which is position. Measure these:

Term	Meaning	Where to read it
S	Log position of one complete snapshot set	ClusterTool recovery-plan
A	Earliest position still physically present in the node’s cluster-log recording	ArchiveTool describe
P	End of log locally available to the restarting node (append position)	ClusterTool recovery-plan / archive descriptor
L	Earliest position the chosen live source can replay or replicate	ArchiveTool describe on that source
C	Proposed new recording start after purge (the cutoff)	your purge plan

A complete snapshot set is one consensus-module snapshot (serviceId = -1) plus one snapshot for every configured service, all at the same leadership term and log position. A set missing one service is not a fallback — it is nothing.

Two invariants follow directly.

Local snapshot recovery works when:

A ≤ S ≤ P

A stale node can catch up when:

L ≤ P

And because purge operates on segment boundaries, the usable local cutoff is:

C = greatest valid segment boundary such that C ≤ S

Never round C up past S. Local recovery starts its replay at the snapshot position; take the log at S away and the snapshot you carefully retained becomes unusable.

Key Insight: L > P is a decision, not an error message

When L > P there is nothing to debug. The node’s next byte does not exist anywhere it can reach, and restarting it repeatedly just burns incident time before you rebuild anyway — often after hours of replay that ends at the gap. Compute L and P first, decide once, and commit to the branch.

Pick a retention model, and write it down

The invariants above have a policy consequence that most teams leave implicit until an incident forces it. Choose deliberately:

Model A — any configured member may restart normally. For every member you retain a complete verified snapshot set, its subsequent local log, and enough log on the live source to bridge from that member’s last verified append position. You never purge past an offline member’s recovery position. This caps how aggressively you can reclaim disk, and one long-offline node holds the whole cluster’s retention hostage.

Model B — nodes outside the window are rebuilt. You retain enough for recent restarts, run a continuously verified ClusterBackup, and let older log go. A node outside the window is reclassified as a rebuild, not a restart.

Model B is the right default for replaceable cloud instances, and it is the honest one: it makes the backup load-bearing and forces you to test it. Model A quietly promises normal restart for a node nobody has checked in weeks.

Model B’s promise is only as good as your last restore drill

Model B trades log retention for a backup dependency, so the backup stops being a safety net and becomes the primary recovery path for any stale node. A ClusterBackup that has never been restored onto an empty host is not a recovery plan — it is an untested assumption with a cron job. Drill the blank-host restore, measure it, and record the date alongside your retention config.

Before you start a stale node: the go/no-go gate

Starting a node because its files exist is how a 20-minute rebuild becomes a 4-hour outage. Run this first, with the node stopped.

1. Collect the facts

AERON_ALL_JAR=/opt/aeron/aeron-all.jar
CLUSTER_DIR=/var/lib/my-cluster/cluster
ARCHIVE_DIR=/var/lib/my-cluster/archive
SERVICE_COUNT=1

java -cp "$AERON_ALL_JAR" io.aeron.cluster.ClusterTool "$CLUSTER_DIR" recording-log
java -cp "$AERON_ALL_JAR" io.aeron.cluster.ClusterTool "$CLUSTER_DIR" recovery-plan "$SERVICE_COUNT"
java -cp "$AERON_ALL_JAR" io.aeron.archive.ArchiveTool  "$ARCHIVE_DIR" describe

Record S, A, P, L, plus H (current cluster commit position) and B (position of the newest verified backup image). Then confirm every recording the recovery plan selects actually exists, the snapshot set covers the consensus module and every service, the local log is contiguous over the range to be replayed, and member/cluster ID, service count, application version, and endpoints are correct.

ArchiveTool verify mutates

verify can mark faulty catalog entries INVALID. Copy the stopped node’s storage before running it, and treat it as a diagnostic mutation rather than a read. Never run offline archive tools against a live archive.

2. Cost each valid path

local replay work           = P - S     (snapshot exists)
full local replay work      = P - A     (no snapshot)
leader transfer / catch-up  = H - P
backup image tail catch-up  = H - B

These are byte ranges, not times — local service replay is usually CPU/application-bound while archive replication is storage- or network-bound, so you need measured throughput from your own restore drills to convert them. Keep those numbers: replay positions/sec, restore bytes/sec, replication bytes/sec.

The newest base is not automatically the fastest. A node 20 minutes stale whose replay runs at application speed can lose to a backup image that is an hour old but restores at disk speed. Choose the lowest total of preparation + replay + transfer + verification.

3. Reject the normal-start path outright when

the selected snapshot recording is unavailable
or the local replay range is not contiguous
or L > P
or no snapshot exists and A does not represent valid initial state

4. Set an abort threshold before you start

Decide the give-up condition in advance, because mid-incident optimism is what consumes the window:

• no recovery-position progress for one configured recovery timeout;
• repeated return to the same election state;
• a requested replay position before the source recording start;
• archive reports an unknown recording or unavailable segment;
• the projected completion estimate exceeds your RTO.

While a recovery is in flight, pause purge jobs, avoid recovering two followers against the same leader archive at once, and don’t throttle the replication path you are depending on.

5. Watch the right signals

Track election state, local replication and replay positions, append position, commit position, the remaining gap H − local, and archive/cluster error counters — see Monitoring and Alerting for the counters and alert rules. Success is FOLLOWER_READY plus position convergence plus application-state validation, not merely a process that is still alive.

The three follower states you will pass through, in order:

Which stages actually run depends on the node’s positions — a current node may skip straight to catch-up; a stale one does all three.

The safe purge workflow

purgeSegments enforces exactly two things, both documented in its own javadoc: the new start position “must be first byte position of a segment after the existing start position”, and “it is not possible to detach segments which are active for recording or being replayed.” Neither of those is a recovery check. The workflow below is.

Phase 1 — establish a recovery point

Trigger a cluster snapshot, wait for completion on every member that must support normal local restart, and verify a complete set on each. Record each snapshot position and recording ID. Then verify the snapshot recordings can actually be replayed — presence in recording.log is metadata, not proof. Do the same for the backup and record its position.

Phase 2 — calculate the boundary

For each member take the oldest retained complete snapshot position S_member; for each offline member promised a normal restart, its last verified append position P_member.

C_live   ≤ min(P_member for every member promised normal catch-up)
C_member ≤ S_member

Round down to a valid segment boundary. If an offline node falls behind the proposed cutoff, make the choice explicitly: postpone the purge, or withdraw the promise of normal restart and mark that node for rebuild. Do not let the arithmetic decide silently.

Phase 3 — dry run

Produce a purge manifest before touching storage: cluster and member ID, cluster-log recording ID, current start/stop positions, proposed new start, retained snapshot sets, the oldest normal-restart member position, backup position and last successful restore-test date, and the exact segment files expected to become detachable.

Reject the purge if no complete local snapshot set is retained, the cutoff is after S, a protected stale node sits before C_live, the backup is absent or untested under Model B, any selected recording is active or being replayed, or the catalog, recording.log, and physical recordings disagree.

Phase 4 — execute conservatively

Keep quorum available throughout, and go one node at a time:

1. stop one follower cleanly;
2. back up its cluster metadata and archive catalog;
3. detach segments to the calculated boundary;
4. validate the archive and the recovery plan;
5. delete the detached segments only after validation passes;
6. restart the follower and wait for FOLLOWER_READY;
7. verify service state and position convergence, then move to the next follower.

Handle the leader last, after leadership has moved or inside a controlled maintenance window.

Key Insight: detach and delete are separate API calls — use both

AeronArchive exposes detachSegments, deleteDetachedSegments, and the combined purgeSegments. The combined call is one irreversible step; the split pair gives you a validation window where the catalog start position has moved but the bytes are still on disk, so a failed recovery-plan check is recoverable. Prefer detach → validate → delete for scheduled maintenance and keep purgeSegments for cases where you have already proven the boundary.

Never use filesystem deletion as the purge mechanism. Go through the archive APIs so the catalog and the recording start position stay coherent — rm on segment files leaves the catalog describing data that is gone.

Phase 5 — prove recovery

Reclaimed disk is not success. Before retiring the previous generation, test: restart of a current node; restart of a node at the oldest supported stale position; fallback to the previous retained snapshot; rebuild of a blank host from backup; and application-state equality after catch-up.

What may actually be deleted

Artifact	Rule
Cluster-log segments	Detachable before a calculated safe boundary, subject to the invariants above. Never purge the whole recording while recording.log references it.
Snapshot recordings	Only complete generations older than every retained fallback, unreferenced by the intended recovery plan and by any restore procedure. Retain two complete generations where storage allows. Never keep only some services of a generation.
Orphaned snapshot recordings	Recordings from a failed snapshot that never reached recording.log are reclaimable once you confirm no recovery metadata references them.
recording.log	Recovery metadata, not bulk data. Do not purge it alongside segments, and do not hand-edit it.

seed-recording-log-from-snapshot is not a retention command

It reads better than it behaves. Per its own javadoc it creates a new recording log from the latest valid snapshot “whose term base and log positions are set to zero”, backing the original up as recording.log.bak. That rebasing is the point — and it is why it belongs only inside a tested migration or restore procedure, never in a retention cron job. Rebase the leader and you have deleted the full-log-from-zero rebuild path for every future blank node in the cluster.

Automate on positions, not on disk usage

A time or free-space policy may propose a cutoff; only the positional checks may approve it:

candidate cutoff = segment boundary constrained by
                     retained local snapshot positions,
                     protected stale member positions,
                     backup / rebuild policy,
                     active recording and replay status

Alongside the cluster counters, track archive recording start and stop positions, the latest two complete snapshot positions per member, backup state and age, projected disk exhaustion, and the date of the last successful blank-node restore test. That last one is the metric that tells you whether Model B is real.

For an EC2 deployment specifically: put archive and cluster state on persistent storage if instance replacement should preserve local recovery; otherwise declare instances disposable and make backup provisioning part of bootstrap. Run at least one continuously verified ClusterBackup outside the cluster’s failure domain. And never reuse a member ID on an empty host unless full-log-from-zero recovery is intentionally supported or that host has been provisioned first.

How this moves your numbers

• RTO is set almost entirely by which recovery base you pick, and that choice is constrained by what you purged weeks earlier. P − S (replay distance) is the number you control by snapshotting more often; H − B is the one you control with backup freshness.
• p99 / max latency is touched only indirectly, but the link is real: Phase 1 requires a cluster snapshot, and in OSS every member snapshots at the same log position, so a retention cycle imposes a stop-the-world pause on the hot path. On a busy matching engine that lands on max latency — which is the argument for standby snapshots if you are on Premium, and for scheduling purge windows off-peak if you are not.
• Throughput is unaffected by purge itself, but a recovering follower competes with live traffic for the leader’s archive replay, disk, and network. Recovering two at once is how a recovery becomes a throughput incident.

Sources

• Purge semantics quoted verbatim from AeronArchive.java — purgeSegments, detachSegments, deleteDetachedSegments javadoc (“must be first byte position of a segment after the existing start position”; “not possible to detach segments which are active for recording or being replayed”), with server-side enforcement in ArchiveConductor.purgeSegments (isValidDetach / isDeleteAllowed).
• Follower catch-up states from ElectionState.java (FOLLOWER_LOG_REPLICATION(9), FOLLOWER_REPLAY(10), FOLLOWER_CATCHUP(13), FOLLOWER_READY(16)) and Election.java; startup recovery from local metadata in ConsensusModuleAgent.java and recovery-plan construction/validation in RecordingLog.java.
• seed-recording-log-from-snapshot rebasing to zero and the recording.log.bak backup, plus the recording-log / recovery-plan / invalidate-latest-snapshot commands, from ClusterTool.java. Archive-side tooling (describe, verify, compact, delete-orphaned-segments, mark-invalid) from ArchiveTool.java.
• Operational guidance: Cluster Backup and Operating Aeron Cluster.
• Background: The Aeron Files.

Members only

The rest of this page is for approved members.

Still on this page:

1. The one question this scenario turns on
2. The five positions
3. Pick a retention model, and write it down
4. Before you start a stale node: the go/no-go gate
5. The safe purge workflow
6. What may actually be deleted
7. Automate on positions, not on disk usage
8. How this moves your numbers
9. Sources

Log in or request access

Sign in with an emailed link, no password. New accounts are approved by hand.