--- title: "21 — Reliability and Resilience: Bulkheads, Circuit Breakers, Retries, and HA" uid: reliability-and-resilience tags: ["resilience", "circuit-breaker", "bulkhead", "high-availability", "reliability", "roadmap:system-design", "system-design"] excerpt: "Reliability patterns (bulkhead, circuit breaker, retry, compensating transaction) absorb failures locally; availability patterns (stamps, geodes, throttling, health endpoints) keep the system up at the architecture level." date: 2026-08-13T03:27:30+0000 source: https://www.aveshina.my.id/en/blog/reliability-and-resilience --- "Just retry on failure" was my resilience strategy, and it handled one pattern out of many. Writing it down separated the techniques cleanly: **reliability patterns (bulkhead, circuit breaker, retry, compensating transaction) absorb failures locally so they don't cascade, and availability patterns (deployment stamps, geodes, throttling, health-endpoint monitoring) keep the system up at the architecture level.** [1][2] Together they are how a distributed system stays standing. The framing that landed is that failures are not exceptional; they are the steady state in cloud systems — multi-tenant hosting, shared platform services, commodity hardware, and the public internet all but guarantee transient and permanent faults [3]. The question is not _how do I prevent failures_ but _how does the system behave when they happen_. Resilience is the ability to gracefully handle and recover from those failures [3], and the patterns below are the named techniques for it. ## Resilience patterns: contain failures locally These patterns stop a failure in one component from dragging down everything that talks to it [1]. - **Circuit breaker** handles faults that take a variable amount of time to recover from, when connecting to a remote service or resource [4]. The idea is a switch with three states: _closed_ (requests flow normally), _open_ (requests fail fast — the downstream is broken, so don't even try), and _half-open_ (a limited number of test requests probe whether the downstream has recovered). Without a circuit breaker, every call to a failing service waits for a timeout, and the waiting threads pile up until the caller itself collapses — a cascade. The breaker breaks the cascade by failing fast. - **Bulkhead** isolates elements of an application into pools so that if one fails, the others keep functioning [5]. Named after a ship's hull — if one compartment is breached, only that section floods, and the ship stays afloat. In practice this means separate thread pools, connection pools, or resource quotas per dependency or per tenant, so a slow downstream or a noisy neighbor cannot consume everything. - **Retry** handles transient failures by transparently retrying a failed operation [6]. Effective only for transient faults, and must always be paired with exponential backoff (and ideally jitter) to avoid retry storms — the antipattern where synchronized retries multiply traffic during an outage. - **Compensating transaction** undoes the work performed by a series of steps that together define an eventually-consistent operation, if one or more steps fail [7]. Operations following the eventual-consistency model are common in cloud workflows; when a multi-step saga cannot complete, each completed step needs a compensating action to reverse it, so the system ends in a consistent state rather than a half-done one. - **Health endpoint monitoring** implements functional checks accessible through exposed endpoints at regular intervals, letting external tools verify that applications and services are performing correctly [8]. The backbone of any failover system — the load balancer or orchestrator polls the health endpoint and stops sending traffic to instances that fail. ```figure Circuit Breaker Closed requests flow Open fail fast Half-Open probe limited stop cascading failures by failing fast Bulkhead pool A pool B ✗ pool C pool D one pool fails, the rest keep serving isolate by dependency or tenant Also in the resilience kit Retry — transparently retry transient failures, with exponential backoff Compensating Transaction — reverse completed steps when a saga fails Health Endpoint Monitoring — expose checks for failover decisions Leader Election — pick one instance to run the singleton job ``` ## Availability patterns: stay up at the architecture level Where resilience patterns contain failures within a deployment, availability patterns shape the deployment itself to stay up through whole-region events [2][9]. - **Deployment stamps** provisions, manages, and monitors a heterogeneous group of resources to host workloads or tenants — each copy is a "stamp" (a.k.a. scale unit or cell). Multiple stamps scale the solution almost linearly and separate customer data, improving scalability and allowing deployment across multiple regions [10]. - **Geodes** deploys a collection of backend services into a set of geographical nodes, each able to service any request for any client in any region, in an active-active style — improving latency and increasing availability by distributing request processing around the globe [11]. - **Throttling** controls the consumption of resources used by an instance, tenant, or service, so the system continues to function and meet SLAs even under extreme load [12]. The proactive alternative to letting a noisy neighbor or a traffic spike degrade everyone. - **High availability** balances high resilience, low latency, and cost across geographies, regions, and availability zones, which limit the blast radius of a failure [9]. There is always a trade-off — three nines is cheap, five nines is not, and the gap is architecture. The idea connecting these to the consistency/availability notes: HA is the architectural spend that buys the nines, and the resilience patterns are what keep each of those nines from being eaten by cascading local failures. ## How I use this The habit is to assume every dependency will fail and ask, for each one, what happens when it does. For remote calls, I add a circuit breaker so a slow downstream cannot stall my threads, and a retry with exponential backoff for transient faults. For shared resources, I partition them into bulkheads so one slow consumer cannot starve the others, and I add throttling to protect against traffic spikes. For multi-step sagas, I design a compensating transaction for each step before I ship it, because inventing one during an incident is too late. And I expose a health endpoint from every service so the orchestrator can route around dead instances. The discipline is to wire these in proactively — a system without a circuit breaker on its outbound calls is a system waiting to cascade, and the cheapest time to add resilience is before you need it, not after an outage has shown you where it was missing. ## References [1] Microsoft, "Reliability patterns," Azure Architecture Framework. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns](https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns) [2] Microsoft, "Availability patterns," Azure Architecture Framework. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#availability](https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#availability) [3] Microsoft, "Resiliency patterns," Azure Architecture Framework. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#resiliency](https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#resiliency) [4] "Circuit breaker design pattern," Wikipedia. [Online]. Available: [https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern](https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern) [5] Microsoft, "Bulkhead pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead](https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead) [6] Microsoft, "Retry pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/retry](https://learn.microsoft.com/en-us/azure/architecture/patterns/retry) [7] Microsoft, "Compensating Transaction pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction) [8] Microsoft, "Health Endpoint Monitoring pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring](https://learn.microsoft.com/en-us/azure/architecture/patterns/health-endpoint-monitoring) [9] Microsoft, "High availability patterns," Azure Architecture Framework. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#high-availability](https://learn.microsoft.com/en-us/azure/architecture/framework/resiliency/reliability-patterns#high-availability) [10] Microsoft, "Deployment Stamps pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp](https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp) [11] Microsoft, "Geodes pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/geodes](https://learn.microsoft.com/en-us/azure/architecture/patterns/geodes) [12] Microsoft, "Throttling pattern," Azure Architecture Center. [Online]. Available: [https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling](https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling) [13] DZone, "Resilient microservices — bulkhead pattern." [Online]. Available: [https://dzone.com/articles/resilient-microservices-pattern-bulkhead-pattern](https://dzone.com/articles/resilient-microservices-pattern-bulkhead-pattern) ```quiz Q: A circuit breaker's "open" state means… - requests flow normally to the downstream - requests fail fast without contacting the broken downstream correct: 1 explain: In the open state, the breaker assumes the downstream is broken and fails immediately, preventing threads from piling up on timeouts — which stops cascading failures. Q: Why pair retries with exponential backoff? - to make retries faster on each attempt - to avoid retry storms, where synchronized retries multiply traffic during an outage correct: 1 explain: Without backoff, every client retries at once and traffic multiplies, worsening the outage. Exponential backoff (with jitter) spaces retries out. Q: The Bulkhead pattern is named after a ship's hull because… - it speeds up request handling like a streamlined hull - compartmentalization isolates a breach so only one section floods and the ship stays afloat correct: 1 explain: Bulkheads partition resources (thread pools, connection pools) so a failure in one pool cannot consume the others — one flooded compartment does not sink the ship. Q: Deployment stamps and geodes both primarily improve… - single-region throughput only - availability and latency by deploying across multiple regions/stamp copies correct: 1 explain: Both deploy multiple copies (stamps, geographic nodes) to scale nearly linearly, separate tenants, and keep serving through a regional failure — the architectural layer of high availability. Q: A compensating transaction is used to… - speed up the original transaction - reverse the work of completed steps when an eventually-consistent saga cannot finish correct: 1 explain: In an eventually-consistent multi-step operation, if one step fails, a compensating transaction undoes the completed steps so the system ends in a consistent state rather than half-done. ```