Operations & Resilience

Tuning gets you speed; operations keep you alive. This section is the runbook — what breaks, what it looks like, and exactly what to do about it.

The critical threshold is quorum

For a 3-node cluster, losing 1 node is fine (2/3 quorum holds). Losing 2 is catastrophic (1/3, no quorum). This is why node placement across AZs matters: never put 2 of 3 nodes in the same AZ. Spread them — 1 node per AZ for a 3-node cluster.

In this section

Understand the machinery

• Aeron® Cluster and Raft Consensus — the consensus model, the five safety guarantees, and how a follower actually rejoins.
• Client-Cluster Communication Flow — ingress, egress, sessions, and what the client sees during a failover.
• Archive Durability and Replay — sync levels, replay, live tailing, replay merge.

Design the deployment

• Cluster Placement: Choosing an AZ Topology — the tiered topologies and their RPO/RTO.
• Cluster Standby and Multi-AZ HA Design — Premium HA without the latency penalty: standby snapshots, in-place identity switch.
• Leader Placement and Preferred-Leader Control — pinning leadership where you want it.
• Scaling Aeron® Cluster — what scales, what doesn’t, and why 3 nodes is usually the answer.

Change it while it’s running

• Migrating a Live Aeron® Cluster Across Regions — node-by-node vs whole-cluster swap, with measured downtime.
• Tuning Cluster Failover Time and Tuning Graceful Step-Down Time — collapsing the election and repoint budgets.

When it breaks

• Failure-mode runbook — nodes, network, disk · snapshots and determinism · clients, ops, resources · multi-AZ and Aeron® edge cases.
• Snapshot Failure and Recovery — killed mid-snapshot, corrupt snapshots, invalidate-latest-snapshot, wipe-and-replay.
• Backups and Disaster Recovery — DR by blast radius, and Cluster Backup (OSS) vs Cluster Standby (Premium).
• Archive Retention and Purge — how much log you can delete without destroying a recovery path.

Failure-mode quick reference

Scenario	Data loss	Availability	Action
Single follower crash	None	None (quorum held)	Restart; auto-catchup — snapshot recovery
Single leader crash	Uncommitted msgs	Brief (election)	Wait for auto-election; restart node — tune the failover
Minority crash	None	None	Restart ASAP to restore fault tolerance — infrastructure runbook
Majority / quorum loss	Uncommitted msgs	Total	Manual: restore quorum, then snapshot + log — multi-AZ runbook
All nodes crash	Possible	Total	Cold-start; replay snapshot + log; else external backup
Node too stale to catch up	None	None (it stays out)	Compare L vs P; rebuild rather than retry — retention and purge

Self-healing vs. manual

Minority failures self-heal — Aeron handles them automatically. Majority failures require manual intervention. Knowing which is which under pressure is the whole game.