The Server That Fell at 7:07 PM: High Availability, Fault Tolerance and Failover Explained
One SnackNow server fails at 7:07 PM and takes the business with it. Rebuild the incident with redundancy, N+1 capacity, health checks, failover, graceful degradation, and safe failback.
One failed server exposes SnackNow's only path, forcing the team to design real redundancy and controlled failover.
One Failed Server, One Unavailable Business
The previous incident established the promise: SnackNow must turn one successful payment into one correct, durable order and communicate the result. At 7:07 PM, while Riya is placing another order, the investigation gets less subtle. Server A stops responding. SnackNow has one application server, so the load balancer has no healthy destination. The menu freezes, checkout requests time out, and the business becomes unavailable. The promise itself is defined in System Reliability Explained: MTBF, MTTR and SLA, which separates reachability from a correct end-to-end result before redundancy is introduced.
Meera: 'Why did one machine take down the entire business?'
Aman: 'Because that machine was not merely a server. It was the only path.'
The visible symptom is total request failure. The technical cause is a single point of failure, or SPOF: one component whose loss makes the critical service unavailable. Its blast radius is the whole application path because every customer request depends on Server A. Fast responses before 7:07 PM do not help; performance describes how the system behaves while the only path is healthy, not whether another path exists when it disappears.
A single point of failure is any one component whose loss can break the service promise by itself.
Aman's immediate design decision is to remove the one-server dependency. The consequence should be a smaller blast radius: one failed instance instead of one failed business. The tradeoff is cost and operational complexity because another component must be deployed, kept healthy, given state, and tested. At work, trace one critical request and ask, 'What single removal stops every successful path?' Memory callback: one locked shop door means every customer is blocked even if the kitchen behind it still works.
Redundancy: Give the System Another Path
Redundancy means providing additional components or paths so the required function can continue after a defined failure. SnackNow adds another application instance, but Aman refuses to call the job complete. A spare counter helps only if it can actually serve the next customer.
Useful redundancy can exist at several layers:
Compute: multiple application instances can execute the same critical request.
Database: a suitable replica or standby can preserve data service when the primary fails.
Storage: critical bytes are not held on one device or one failure domain.
Network: traffic has another route instead of one switch, link, or gateway.
Availability zones: replicas do not all disappear with one building-scale infrastructure failure.
The story event is Meera pointing to two server icons and asking whether SnackNow is now safe. The visible risk is that the second instance may be unhealthy, too small, unreachable, or missing the session and order state needed to continue. The technical cause of false reassurance is redundancy without a usable failover path.
Aman defines five conditions: the spare must be healthy; remaining instances must have enough capacity; the load balancer must be able to reach them; required state must be externalised, replicated, or reconstructable; and the switch must be possible within the service target. The consequence is functional redundancy. The tradeoff is that every extra path adds monitoring, testing, and data-consistency work. In a workplace review, do not count replicas. Remove one component on paper and trace whether the request still completes correctly.
Reader question: Why can a diagram with two servers still contain a single point of failure?
Why this visual is needed: Counting server boxes hides shared dependencies and missing traffic paths.
How to read it: Compare the one-path design, the duplicate-looking design with one shared failure source, and the design whose alternate path is independently reachable and ready.
What to remember: Redundancy is another usable path, not another rectangle.
N, N+1 and More Than One Spare
At the 7 PM peak, measurements show that one SnackNow server can safely process 50 checkout requests per second while preserving latency and error targets. The expected demand is 100 requests per second. That means N, the capacity required for normal operation, is two servers.
N = required capacity = 2 servers
N+1 = 3 deployed servers
After one failure: 2 healthy servers remain
Peak requirement: still satisfiedIf SnackNow deploys exactly two servers and runs both near 100%, the system has two instances but no failure headroom. When Server A fails, Server B receives twice its safe load. Queues grow, latency rises, timeouts trigger retries, and the surviving server may collapse. The visible symptom changes from a clean node failure into overload and possibly a cascading outage.
N+1 means deploying the N components needed for the workload plus one spare unit of the same relevant capacity. Here, three servers allow one to fail while two still carry peak demand. N+2 briefly means protection against two comparable component losses, but only if dependencies, traffic routing, and state also survive. It is not a magic label for every two failures.
The design decision is to reserve failure capacity instead of treating normal saturation as efficiency. The consequence is that failover does not immediately create a performance incident. The tradeoff is underused capacity during healthy periods. Autoscaling may replenish capacity, but it is not instantaneous protection; the surviving fleet must bridge detection and startup time. At work, calculate capacity after the failure, not before it. Memory callback: the spare counter matters because two counters are required for the queue even on an ordinary evening.
Reader question: If SnackNow needs two servers at peak, how many should be deployed to survive one server failure?
Why this visual is needed: N+1 is often recited without testing the remaining capacity against real demand.
How to read it: Remove one server from each lane, then compare what remains with the demand line.
What to remember: N is what the workload needs; plus one is failure headroom.
Active-Active: Both Counters Work
SnackNow can use its redundant instances in different ways. In an active-active design, both Server A and Server B accept live traffic during normal operation. The load balancer distributes Riya's request to one healthy instance, while other orders go to the other. When Server A stops responding, the load balancer removes it and the already-active Server B absorbs more traffic.
The visible benefit is high utilisation and a short transition because the surviving node is already serving real requests. This works only when request handling is compatible with multiple active nodes. Sessions cannot live only in Server A's memory unless Riya is deliberately pinned and that loss is acceptable. Order writes need idempotency, shared or coordinated state, and database rules that prevent two active workers from producing conflicting outcomes.
The design decision is to externalise session state, make order commands safe to retry, and provision each active node so the remaining fleet can take the failed node's share. The consequence is rapid failure absorption and capacity use during healthy operation. The tradeoff is deeper coordination: concurrent writers, cache invalidation, deployment compatibility, and conflict handling become real engineering work.
At work, active-active is not shorthand for 'put two servers behind a load balancer.' It is a claim that both can safely process production work at the same time. The earlier load balancing walkthrough explains the traffic director; here the reliability question is whether every destination can complete the same request correctly. Memory callback: two working counters serve customers before anything fails.
Active-Passive: One Works, One Waits
In an active-passive design, Server A is the primary and Server B is the standby. Normal traffic reaches Server A. Server B waits in a prepared state until health logic or an operator promotes it. The empty backup counter is not useless; its job is to be ready when the working counter disappears.
The approach can simplify ownership of state because only one node is authoritative at a time. That can reduce concurrent-write conflict, which is why active-passive arrangements are common around stateful systems. But the standby may need boot time, configuration, current data, credentials, and a promotion step. A cold standby can take much longer than a warm one, while a hot standby costs more and behaves more like live infrastructure.
The story symptom during failover may be a short interruption while SnackNow confirms Server A is truly unavailable and promotes Server B. The design decision is to define promotion authority, data freshness, capacity, and a tested switch procedure. The consequence is a simpler active state model. The tradeoff is paid but underused capacity plus potentially longer recovery.
Decision question | Active-active | Active-passive |
|---|---|---|
Who handles normal traffic? | Every active node | Primary only |
What happens after failure? | Survivors absorb traffic | Standby is promoted |
Main engineering burden | Concurrent state and coordination | Promotion readiness and recovery time |
Capacity value while healthy | Used for live work | Reserved for failure |
Good fit when | Work is safely distributable | Single authority simplifies critical state |
Table takeaway: Choose based on state, recovery target, and failure capacity, not because one pattern sounds more advanced.
Reader question: What changes for the same order when both nodes work versus one node waiting?
Why this visual is needed: A feature list does not show when traffic and state move.
How to read it: Follow Riya's order in both lanes and notice whether traffic is shared before failure or transferred after failure.
What to remember: Active-active shares work before failure; active-passive transfers work after failure.
High Availability Is Not the Same as Fault Tolerance
Meera now asks the interview question that causes the most confusion: if both designs survive Server A, are high availability and fault tolerance the same? Aman replays the identical failure twice so the difference remains visible.
The High-Availability Version
Server A fails at 7:07:00. Two failed health checks later, the load balancer removes it. Requests in flight may fail or retry. At 7:07:12, Server B carries the traffic. SnackNow experienced a brief interruption, but the architecture minimised downtime and restored service within its target. That is high availability: the service is designed to remain usable for a high proportion of time through redundancy, detection, and recovery.
The Fault-Tolerant Version
Now imagine the system is specifically designed to tolerate one application-node failure without a user-visible interruption. Work is replicated or safely retried, enough active capacity already exists, state is accessible, and routing shifts without losing Riya's order. Server A still fails, but the covered failure does not interrupt the required function. That is fault tolerance for this defined failure class.
High availability minimises interruption; fault tolerance aims to mask a defined failure while service continues.
How is it so far?
Vote with other readers
The exact scope matters. SnackNow might tolerate one stateless node failure but not a database corruption, shared credential expiry, or whole-region loss. Saying 'the system is fault tolerant' without naming the tolerated faults is incomplete.
The design decision depends on business impact. Briefly retrying a restaurant-review request may satisfy HA, while a payment authorisation step may justify stronger masking and duplicate prevention. The consequence of fault tolerance can be less user-visible disruption. The tradeoff is higher cost, tighter state coordination, and more complex testing. In a workplace design review, state both the interruption target and the failure class. Memory callback: HA reopens the counter quickly; fault tolerance keeps the covered order moving.
Reader question: What does Riya notice when HA and fault-tolerant designs face the same node failure?
Why this visual is needed: Different analogies make both terms blur together.
How to read it: Start from the same failure marker and compare the user journey, not the number of replicas.
What to remember: HA shortens interruption; fault tolerance masks a scoped failure.
Health Checks: Who Notices the Server Is Dead?
Redundancy cannot help until something decides where traffic may safely go. SnackNow gives each instance a health light, immediately mapped to a health check: an automated probe or signal used to judge whether the instance should receive traffic.
A liveness check asks whether the process is alive enough to keep running. A readiness check asks whether it is prepared to serve requests now. Restarting a temporarily unready process can make an incident worse, while routing orders to a live-but-unready process can harm customers.
A shallow check might return HTTP 200 because the web process can answer `/health`. But suppose Server A cannot reach the order database. Its process is alive, yet it cannot complete checkout. A dependency-aware readiness check should reflect critical capabilities without turning every optional dependency into a reason to remove the instance. If recommendations are down but checkout works, marking the whole server unready would discard useful capacity.
Signal | Question | SnackNow action |
|---|---|---|
Liveness | Should this process be restarted? | Restart only when the process cannot recover itself |
Readiness | Should this instance receive new orders? | Remove from routing until the critical path is ready |
Dependency-aware | Can it perform the promised journey? | Check critical dependencies without coupling to optional features |
Intervals and thresholds create a tradeoff. One failed probe can be a transient network delay; waiting for ten failures can extend the outage. A false positive removes a healthy server and reduces capacity. A false negative keeps sending traffic to a broken server. The design decision is to use representative checks, multiple observations, timeouts, and separate liveness from readiness. The consequence is faster, safer detection; the tradeoff is probe load and threshold tuning. At work, test a process that is alive but database-isolated. Memory callback: a green health light must mean ready for this traffic, not merely running.
Failover: Moving Traffic Without Losing the Story
Failover is the controlled transfer of work from a failed or unhealthy component to a healthy alternative. It is a sequence, not a magic arrow.
Server A fails or becomes unable to serve the critical path.
Health checks cross the failure threshold and identify the condition.
The load balancer stops assigning new requests to Server A.
Server B receives traffic, with enough capacity to preserve the target.
SnackNow verifies that session, order, and data state remain safe.
Monitoring opens the incident workflow and humans inspect the cause.
For application servers, the load balancer can perform the traffic switch in seconds. DNS-based failover can redirect a hostname to another endpoint, but resolver caching and TTL behaviour make timing less direct. Database failover involves promoting a suitable replica or standby and ensuring applications use the new writer; the replication and sharding article gives the data foundations, while this article stays focused on availability behaviour. Failover changes the serving system, but it cannot rewind a logical deletion; Backup and Recovery Strategies: RPO, RTO and PITR covers the historical recovery path.
Automatic failover is fast and useful for well-understood failure signals. Manual failover gives operators more context when a wrong promotion could corrupt or split state. Neither is universally safer. SnackNow automates removal of an unhealthy stateless instance but makes database promotion obey stricter fencing and data checks.
Riya's session cannot exist only in Server A's memory. SnackNow either uses an external session store, a signed stateless token for appropriate claims, or a deliberate affinity strategy with a clear failure consequence. The design decision is to make critical state survive the traffic move. The consequence is continuity; the tradeoff is an external dependency or more careful token and consistency design. In the workplace, rehearse failover with an in-flight request, not an idle dashboard. Memory callback: the traffic arrow must carry the order story, not just the next HTTP request.
Failback: Returning Home Can Cause Another Outage
At 7:24 PM, engineers repair Server A. The dangerous instinct is to reverse the arrow immediately and send half the traffic back. But Server A may have stale caches, old configuration, missing warm connections, unsynchronised state, or the same unresolved fault. A repaired label is not evidence of readiness.
Failback is the controlled return of work to the restored original component or normal topology. SnackNow first resynchronises required state, checks version and configuration parity, warms critical dependencies, and proves readiness. It then sends a small canary share of traffic, watches errors and latency, and increases traffic gradually. If the health signal oscillates between good and bad, dampening or hold-down periods prevent flapping from moving traffic repeatedly.
The design decision is to separate repair from reintegration. The consequence is a slower but safer return to full redundancy. The tradeoff is operating temporarily at reduced headroom while verification continues. At work, write failback steps alongside failover steps; an incident is not over merely because the standby works. Memory callback: the replacement counter serves one test customer before the entire queue is moved.
Reader question: What exact sequence turns a dead server into safely restored normal capacity?
Why this visual is needed: One failover arrow hides detection, degraded operation, replacement, verification, and failback risk.
How to read it: Move from the failed health light to traffic removal, protected core service, replacement, readiness proof, canary traffic, and full restoration.
What to remember: Detect, remove, protect, replace, verify, canary, restore.
Graceful Degradation: Keep the Main Promise
After Server A fails, Server B is healthy but operating with less headroom. SnackNow can pretend every feature has equal importance and risk losing checkout, or it can deliberately reduce non-essential work. Meera chooses the second path.
Preserve first | Degrade temporarily | Reason |
|---|---|---|
Menu, cart, checkout | Personalised recommendations | Customers must still form and place an order |
Current order status | Review feed | Existing transactions need trustworthy progress |
Payment and idempotency controls | Animated map tracking | Financial correctness outranks visual fidelity |
Restaurant order delivery | Coupon suggestions | The core fulfilment path must retain capacity |
This is graceful degradation: the system preserves its essential promise while offering reduced fidelity during stress or partial failure. SnackNow disables recommendations and review feeds, serves a simpler tracking state, sheds optional background work, and returns explicit fallbacks instead of letting every dependency timeout.
The technical cause of the risk is shared capacity: optional work competes with the critical path. The design decision is to classify features before the incident and build controllable fallbacks. The consequence is that Riya can still order even though the app feels simpler. The tradeoff is temporary experience loss and more code paths to test. The messaging and queues lesson shows how non-critical follow-up work can be decoupled, but a queue must still have bounded backlog and idempotent consumers.
During failure, preserve the smallest journey that still keeps the business promise.
At work, ask product and engineering to name the first feature they would disable and the last one they would sacrifice. Memory callback: the reduced menu board keeps checkout visible while decorative choices disappear.
Self-Healing: Detect, Replace, Verify
The load balancer removing Server A prevents new harm, but SnackNow is still one failure away from another outage. Self-healing restores the desired healthy capacity for known failure classes. An orchestrator can restart a dead process, terminate an unhealthy instance, reschedule a container or pod, and create a replacement until the desired replica count returns.
The sequence matters. Replacement Server C starts, loads configuration, connects to dependencies, warms what must be warm, and passes readiness checks before receiving traffic. Simply restarting a process does not prove that its database path or credentials work. If automation repeats the replacement loop, alerts must bring humans in before churn consumes capacity or hides a bad deployment.
The story event is a new replacement counter appearing while Server B carries the queue. The visible symptom being corrected is reduced capacity, not the original root cause. The design decision is to automate a bounded, verified repair. The consequence is shorter time at risk and restored N+1 posture. The tradeoff is automation that can reproduce a bad image or configuration at machine speed.
The earlier autoscaling and cloud best-practices article explains adding capacity for demand. Self-healing may use some of the same machinery, but its trigger is unhealthy capacity, not merely rising traffic. In the workplace, test whether the replacement is genuinely ready before the router sees it. Memory callback: self-healing builds the replacement counter; it does not understand why every counter broke.
Redundant but Still Broken: Common-Mode Failures
SnackNow now has three application instances. Meera relaxes until Aman draws one electricity line feeding all three counters. The shared line immediately maps to a common-mode failure: one cause that defeats multiple supposedly redundant components together.
All replicas can fail in the same availability zone. The same bad deployment can crash every instance. One shared database can make three healthy app servers useless. A shared credential can expire everywhere. One configuration bug can make every health check lie. One network path can isolate the entire fleet. Identical copies often share identical weaknesses.
The visible symptom may still be total outage, but the technical cause is correlated failure rather than too few replicas. The design decision is to diversify failure domains where the business target justifies it: spread instances across zones, use staged deployments, isolate configuration changes, protect critical credentials, and test dependency loss. The consequence is reduced correlated blast radius. The tradeoff is more topology, rollout time, and cost.
At work, circle every dependency shared by all redundant paths. Then ask what happens if it fails or becomes wrong. This does not yet teach regional disaster recovery; it establishes why local redundancy has a boundary. Memory callback: three counters on one electricity line can go dark together. When the shared boundary becomes an entire region, Disaster Recovery in System Design: RTO, RPO and DR takes over from local high availability.
Designing SnackNow's Failure-Survival Plan
Aman now rebuilds the 7:07 PM path, with every decision tied to the failure rather than to a tool catalogue.
Design decision | Incident problem solved | Tradeoff accepted |
|---|---|---|
Three app instances across zones | One instance or one zone does not remove all compute | Extra baseline capacity and cross-zone design |
Load balancer with readiness checks | Broken instances stop receiving new orders | Threshold tuning and probe ownership |
N+1 peak capacity | One failed instance does not overload survivors | Paid failure headroom |
Externalised session and idempotent order state | Riya's journey survives a traffic move | Shared-state dependency and consistency controls |
HA database arrangement with fenced promotion | The order path is not tied to one writer process | Replication lag and promotion complexity |
Critical-feature classification | Optional work cannot consume checkout capacity | Degraded modes must be designed and tested |
Automatic replacement plus controlled failback | Desired capacity returns without flooding an unproven node | Automation guardrails and slower reintegration |
Table takeaway: Each control exists because a visible incident symptom exposed a specific missing path; none is free.
During the next rehearsal, Server A is terminated. Readiness fails, the load balancer removes it, Server B and Server C keep the peak within capacity, optional features reduce, and the orchestrator creates Server D. Riya's session and idempotency key remain available. The team watches the canary before restoring the normal traffic split. The consequence is a bounded component incident instead of a business outage.
The tradeoff is operational discipline. Three replicas with untested health checks are less trustworthy than two replicas with clear failure behaviour. In a workplace design, write the expected symptom, detector, automated response, human escalation, and recovery evidence for each failure class. Memory callback: every object now has an engineering job: spare counter, health light, traffic arrow, reduced menu, and replacement counter.
Common Misunderstandings
Claim | Why it sounds right | SnackNow correction | Memory line |
|---|---|---|---|
Two servers automatically mean HA | Another server appears to remove the SPOF | Prove health, capacity, routing, state, and failover | A spare behind a locked door is not a path |
HA and fault tolerance are the same | Both use redundancy | HA may briefly interrupt; FT masks a scoped fault | Reopen quickly versus keep serving |
Active-active is always better | All capacity is used | Concurrent state can cost more than standby time | Two working counters need coordination |
A passive node costs nothing | It handles no normal traffic | It still needs compute, data, tests, patching, and capacity | An empty counter must remain ready |
HTTP 200 means healthy | The process answered a probe | Readiness must represent the critical path | A lit sign cannot prove the kitchen works |
Automatic failover cannot make mistakes | Automation reacts consistently | Bad signals can promote the wrong node or split authority | A fast wrong arrow is still wrong |
Failback just reverses failover | The old primary is repaired | Resynchronise, verify, canary, and prevent flapping | Test one customer before the queue |
Self-healing removes alerts | Automation replaces instances | Repeated or unknown failures still require humans | A replacement is not an explanation |
Redundancy covers every disaster | Many copies look safe | Shared zones, code, credentials, data, and networks can fail together | Check the electricity line |
Table takeaway: Every reassuring phrase must be converted into a failure class, a measurable response, and a known boundary.
Memory Reconstruction: Rebuild the Architecture from the Incident
A. Seven-Scene Incident Strip
7:07 PM: Server A stops responding and the only request path disappears.
Aman identifies the single point of failure and adds usable redundancy.
Capacity planning changes the fleet from N to N+1.
Health checks detect that Server A is not ready for traffic.
Failover moves orders while graceful degradation protects checkout.
Self-healing creates a replacement and restores the desired replica count.
State verification and canary traffic make failback controlled rather than automatic reversal.
Close the article and retell those seven scenes. If one scene disappears, identify the missing engineering control.
B. Memory Objects
Object | Concept | Question it recalls |
|---|---|---|
One locked shop door | Single point of failure | What one loss blocks every path? |
Spare counter | Redundancy | Can the alternate really serve? |
Two working counters | Active-active | Can both safely process live work? |
Empty backup counter | Active-passive | How is the standby promoted? |
Health light | Liveness and readiness | Should this process run and receive traffic? |
Traffic arrow | Failover | Who moves work, and what state follows? |
Reduced menu board | Graceful degradation | What must remain available first? |
Replacement counter | Self-healing | How is healthy capacity restored? |
Shared electricity line | Common-mode failure | What can break every copy together? |
C. Cause -> Choice -> Consequence
Cause: Server A was the only path. Choice: add independently reachable N+1 capacity, representative readiness checks, safe state, and tested routing. Consequence: one node failure becomes a smaller incident, although cost and coordination rise. Cause: the fleet runs with less capacity. Choice: degrade optional features and replace the node. Consequence: checkout survives while experience fidelity temporarily falls. Cause: the repaired node may still be unsafe. Choice: verify and canary failback. Consequence: normal topology returns without causing a second outage.
D. Spoken Retelling
SnackNow went down because one application server was the only path. Redundancy helped only after the alternate was healthy, reachable, state-safe, and large enough. Active-active shared traffic before failure; active-passive promoted a standby after failure. Health checks triggered failover, graceful degradation protected checkout, self-healing restored capacity, and controlled failback returned traffic. The design still had to guard against common-mode failures.
E. Contrast Test
Use the same Server A failure. If Riya sees a short retry while Server B takes over, explain why the design can be highly available. If her covered order continues without a visible interruption, explain why the design may be fault tolerant for one node failure. Then change the fault to a shared database loss and say whether either claim still holds.
F. Remove One Component
Point at the load balancer, one application instance, the session store, the database writer, one zone, and the deployment configuration. Remove each one in turn. For every removal, say the visible symptom, detection signal, alternate path, remaining capacity, state risk, and recovery action. That exercise reveals SPOFs more reliably than counting replicas.
G. Workplace Scenario
A production server fails. The load balancer successfully sends new requests to the second server, but every signed-in user becomes logged out and open carts disappear. What failed even though network failover worked?
A strong answer says session and cart state lived only on the failed server or was otherwise unavailable to the survivor. Traffic redundancy existed, but state continuity did not. The design should externalise or deliberately replicate the required state, choose appropriate consistency and security controls, and test failover with a real signed-in journey. The tradeoff is another shared service or more complex stateless-token design.
Final Mental Model
Return to Server A at 7:07 PM. The important event was not that hardware failed; hardware and processes eventually do. The architectural failure was that SnackNow had no second usable path, no reserved failure capacity, and no state-safe way to move Riya's work.
Design availability by tracing what happens after one component disappears, not by admiring the healthy diagram.
Reliability defined the promise. High availability gave SnackNow another path. Fault tolerance reduced what Riya could notice. Failover moved the work, graceful degradation protected the important journey, and self-healing rebuilt the missing capacity.
Lock in the takeaway
Frequently asked questions
What is a single point of failure in system design?
A single point of failure is one component whose loss can make a required service or critical journey unavailable by itself. It can be a server, database, network path, credential, configuration source, or another shared dependency.
Why do two servers not automatically provide high availability?
Both servers must be healthy, reachable, state-safe, and able to carry the remaining load, while traffic routing and dependencies must also survive. Two replicas that share one fatal dependency or lack a working failover path can still fail together.
What is the difference between high availability and fault tolerance?
High availability minimises downtime and may include a brief interruption during detection and failover. Fault tolerance aims to continue the required function without user-visible interruption for explicitly defined failure classes.
What is active-active vs active-passive?
In active-active, multiple nodes handle live traffic and survivors absorb work after a failure. In active-passive, one primary handles traffic while a standby waits for promotion, usually simplifying active ownership but adding promotion time and reserved capacity cost.
What does N+1 redundancy mean?
N is the capacity needed for the expected workload. N+1 deploys one additional equivalent unit so the system can lose one unit and still retain the required capacity, assuming routing, state, and dependencies also remain healthy.
How do health checks participate in failover?
Health checks provide signals about liveness and readiness. After a configured threshold, routing or orchestration removes an unhealthy instance, directs traffic to healthy capacity, and may start a replacement. Checks must represent the critical path without overreacting to optional dependency failures.
What is the difference between failover and failback?
Failover transfers work to a healthy alternative after failure. Failback returns work to the restored original or normal topology, ideally after state resynchronisation, readiness checks, canary traffic, and safeguards against repeated switching.
What is a common-mode failure?
A common-mode failure is one cause that defeats multiple redundant components together, such as a bad deployment, shared database, expired credential, single availability zone, common network path, or faulty configuration.
