05 — Availability Patterns: Fail-Over and Replication

"Just add more servers" was my availability strategy, and it skipped the structural choices underneath. Writing it down gave me a model I can actually apply: availability is bought with redundancy, and the two structural patterns for spending that redundancy are fail-over (a standby takes over when the active dies) and replication (copies stay in sync so any one can serve). [1][2] Both add cost and complexity, and both leave specific residual risks worth knowing.

The framing that landed is that "high availability" is not a feature a product has; it is a budget you spend. Every "9" of uptime costs real money in duplicate hardware, and the question is always _where_ you place the redundancy and _what failure mode_ it covers. Fail-over covers a whole component dying; replication covers both death _and_ load-spreading, at the cost of keeping the copies consistent [1].

Fail-over: the standby takes over

Fail-over is the pattern where a primary component handles work, a secondary stands by, and the secondary is promoted if the primary fails [1][2]. The two flavors differ in whether the standby does useful work before the failure:

• Active-passive (a.k.a. master-slave). Heartbeats flow between the active and the passive. Only the active serves traffic. If the heartbeat stops, the passive grabs the active's IP and resumes service. The downtime depends on whether the passive was already running ("hot" standby, fast) or has to boot from cold ("cold" standby, slow) [1].
• Active-active (a.k.a. master-master). Both servers serve traffic and spread the load. DNS (for public-facing) or application logic (for internal) must know about both. If one dies, the other just keeps going with the full load [1].

The honest costs the roadmap flags: fail-over adds hardware and operational complexity, and there is a real window where data written to the active but not yet replicated is lost when the active dies [1]. That last point is the one I now design around explicitly — the replication lag between active and standby is the maximum data-loss window on failover.

Replication: copies that stay in sync

Replication is the other redundancy pattern — keeping multiple copies of the same data in different places, so any of them can satisfy a read or take over if another is lost [2][3]. Two structural flavors, with the same master/slave vs master-master names:

• Master-master. Multiple nodes accept both reads and writes, coordinating with each other. Any can take over if another fails. The cost is conflict resolution: if two masters accept conflicting writes to the same key simultaneously, something has to reconcile them [2].
• Master-slave. One master handles writes and replicates them to read-only slaves. If the master dies, a slave is promoted. Simpler to operate, but the master is a write bottleneck and a single point of write failure until promotion completes [2].

Replication is the pattern that turns a single fragile database into something that survives a node loss without losing the data. Paired with fail-over, it is what makes "promote the slave" a real recovery procedure instead of a hope.

Reading availability in numbers

The other thing I had to internalize is what an availability percentage actually costs. Availability is measured in "nines" — 99.9% is three nines, 99.99% is four [4]. Each additional nine is an order-of-magnitude reduction in allowed downtime:

And the math for combining components is unforgiving: components in series multiply their unavailability (two 99.9% components in series give 99.8%), while components in parallel improve it (two 99.9% components in parallel give 99.9999%) [4]. That is the formal justification for redundancy — parallel copies are how you climb the nines ladder.

How I use this

The habit is to make the budget explicit per service. I name the availability target in nines up front, which forces the redundancy question: how many components fail in series (each must be reliable), and where do I need parallel copies to push the combined number up. For stateful services I always pair replication with fail-over and I name the replication-lag window as the maximum data loss on failover — because "high availability" with silent data loss on promotion is not actually the availability the business thinks it bought.

References

[1] Serverion, "Active-passive vs. active-active failover," serverion.com, 2023. [Online]. Available: https://www.serverion.com/uncategorized/active-passive-vs-active-active-failover/

[2] D. Martin, "Replication: availability pattern," system-design-primer (open source), 2024. [Online]. Available: https://github.com/donnemartin/system-design-primer#replication

[3] EnjoyAlgorithms, "Database replication introduction: types and advantages," 2023. [Online]. Available: https://www.enjoyalgorithms.com/blog/introduction-to-database-replication-system-design

[4] EnjoyAlgorithms, "Availability in system design," 2023. [Online]. Available: https://www.enjoyalgorithms.com/blog/availability-system-design-concept/

[5] "Uptime calculator: how much downtime corresponds to 99.9% uptime," uptime.is. [Online]. Available: https://uptime.is/

[6] DesignGurus, "High availability in system design — 15 strategies for always-on systems," 2023. [Online]. Available: https://www.designgurus.io/blog/high-availability-system-design-basics

[7] "Design patterns for high availability: what gets you 99.999% uptime?," YouTube, 2020. [Video]. Available: https://www.youtube.com/watch?v=LdvduBxZRLs