The Server That Fell at 7:07 PM: High Availability, Fault Tolerance and Failover Explained
One SnackNow server fails at 7:07 PM and takes the business with it. Rebuild the incident with redundancy, N+1 capacity, health checks, failover, graceful degradation, and safe failback.
One failed server exposes SnackNow's only path, forcing the team to design real redundancy and controlled failover.
One Failed Server, One Unavailable Business
At 7:07 PM, Server A stops responding while Riya is placing an order. SnackNow has one application server. The load balancer can detect that Server A is unhealthy, but it has nowhere else to send the request. The menu disappears, checkout stops, and the entire business becomes unavailable.
Meera asks, "Why did one machine take down everything?"
Because Server A was a single point of failure: one component whose loss removed the only working path through a critical user journey. The failure itself was small. The blast radius was the whole service.
Fast responses before 7:07 PM did not help once the only path vanished. System Reliability Explained: MTBF, MTTR and SLA defines the customer promise; this article asks how that promise keeps another path when one component fails.
Redundancy Works Only When the Other Path Is Real
A second server is redundancy, but the word alone proves very little. The spare must be healthy, reachable, compatible with current code and configuration, supplied with required state, and large enough to carry its share of traffic after a failure.
Read the visual from left to right. One server is an obvious single point. Two servers behind a shared broken dependency only look safer. Working redundancy uses independent enough paths so that the same failure does not remove every copy.
Redundancy can exist at several layers: application instances, database replicas, storage copies, network paths, power, availability zones, and even people or procedures. Copying one layer while leaving another layer singular simply moves the single point of failure.
A redundant component is useful only when the system can detect failure, reach the alternate path, carry the load, and preserve correct state.
N+1 Means Capacity Survives One Failure
Suppose SnackNow needs two application servers to handle the evening peak. Here, N equals two: the capacity required for normal service. N+1 means three servers are available, so one may fail while two still provide the required capacity.
Two servers each running at nearly 100% are not safe redundancy for a two-server workload. When one fails, the survivor receives twice its previous load and may collapse. Redundancy without headroom can turn one failure into a cascade.
N+2 provides protection against two covered failures or leaves more maintenance headroom, but it costs more. The useful question is not "How many copies do we have?" It is "Which failure set can we survive at expected peak load?"
Design | After one server fails | Tradeoff |
|---|---|---|
N only | Required capacity is no longer available | Lowest cost, no spare capacity |
N+1 | Required capacity remains for one covered failure | One extra unit plus coordination |
N+2 | More failure or maintenance headroom | Higher standing cost |
Active-Active and Active-Passive Use the Spare Differently
In an active-active design, multiple healthy instances serve live traffic. Capacity is used during normal operation, and a load balancer stops routing to an unhealthy instance. The surviving nodes must still have enough headroom. Sessions, writes, caches, and retries must be coordinated so that moving a request does not create conflicting state.
In an active-passive design, one instance serves while a standby waits. The standby may be hot, warm, or require promotion. This can simplify who owns state, but some paid capacity sits idle and failover can take longer.
The diagram compares the same order flow. Active-active spreads live work across both paths. Active-passive keeps a prepared path that receives traffic only after promotion. Neither is automatically better.
Decision | Active-active | Active-passive |
|---|---|---|
Normal traffic | Several nodes serve | Primary serves |
Failure response | Remove failed node and rebalance | Promote standby, then route |
State challenge | Concurrent work and conflicts | Freshness and promotion readiness |
Cost shape | Capacity is used, but coordination is harder | Standby cost may be underused |
SnackNow uses active-active application instances because requests can be distributed, but it may choose a primary-and-standby pattern for a stateful component. The load balancing walkthrough explains the traffic director in more depth.
High Availability and Fault Tolerance Are Different Targets
The terms are sometimes used loosely, so state the failure scope. A high-availability design aims to minimise downtime. Detection and failover may still create a short interruption. A fault-tolerant design continues through a specified failure without a user-visible interruption or incorrect result.
How is it so far?
Vote with other readers
Run the same Server A failure twice. In the HA version, health checks need several failed observations, remove Server A, and move traffic. Riya may retry after a brief error. In the fault-tolerant version, redundant processing and state coordination hide that specific failure from her request.
Fault tolerance is never unlimited. A service might tolerate one application-node failure and still fail when a shared database, bad deployment, credential, zone, or region disappears. Always finish the sentence: fault tolerant against which failure?
High availability is usually cheaper and easier to operate. Fault tolerance can be justified for safety-critical or extremely expensive interruptions, but it demands deeper redundancy, stricter state coordination, testing, and often more latency or cost.
Health Checks Decide Whether a Node Should Receive Traffic
At 7:07 PM, the load balancer needs evidence. A shallow check may prove that the process accepts TCP connections. It does not prove that checkout can reach required dependencies or make progress.
A liveness check asks whether a process should be restarted. A readiness check asks whether it should receive traffic. A startup check gives slow initialization time before normal checks take over. The names come from container platforms, but the decisions apply more broadly.
Consider a server whose process is alive but whose database connection pool is exhausted. Restarting it repeatedly may worsen the overload. Keeping it in rotation may send customers into failure. A useful readiness check represents the capability needed by the routed request without making every optional dependency a reason to remove the node.
Intervals and thresholds create a tradeoff. Aggressive checks detect failure sooner but can eject healthy nodes during a brief network wobble. Slow checks avoid flapping but extend customer impact. Test false positives, false negatives, and the behaviour when many nodes report unhealthy together.
Failover Is a Sequence, Not a Traffic Arrow
Aman writes the failover sequence in the incident log:
The component fails or stops serving a required capability.
Health evidence crosses the defined threshold.
The failed path is removed or fenced so it cannot keep accepting unsafe work.
Traffic or leadership moves to a healthy target with enough capacity.
State, sessions, and in-flight operations are checked or reconciled.
The critical journey is verified and the incident remains open until stability is proven.
Application-server failover can be quick when requests are stateless or carry recoverable identity. Database failover is harder because one writer must not conflict with another and the promoted replica must be sufficiently current. The replication and sharding article explains how copies and partitioned state behave.
Failover does not recover historical truth after a deletion. Backup and Recovery Strategies: RPO, RTO and PITR explains why a healthy alternate can contain the same mistake.
Automatic failover reduces reaction time for well-understood failures, but a wrong health model can automate the wrong decision. Manual failover is slower but may be safer for ambiguous, high-impact state transitions. Many systems combine automatic component failover with human confirmation for broader events.
Inside one application tier, a load balancer can stop routing to an unhealthy instance quickly. DNS-based failover is usually broader and slower because clients and resolvers may cache earlier answers. The switching mechanism must match the failure boundary and recovery target.
Degrade Gracefully While the Critical Path Recovers
During reduced capacity, SnackNow disables recommendations, reviews, animated tracking, and coupon suggestions. It keeps menu, cart, checkout, and current order status. That is graceful degradation: preserve the most valuable journey at reduced fidelity instead of letting optional work consume the remaining capacity.
Load shedding rejects or delays lower-priority work before the system exhausts itself. Cached or simplified responses can help when they are safe. A queue can absorb work only when delay is acceptable and its capacity is bounded; the messaging and queues lesson shows that tradeoff.
Graceful degradation needs product decisions before the incident. Which features can be stale? Which can disappear? Which actions involve money or safety and must fail closed? The reduced menu board is useful only because Meera and Aman agreed on the main promise in advance.
Self-Healing Replaces Known Failures, Then Verifies Readiness
Self-healing can restart a stuck process, replace an unhealthy instance, reschedule a workload away from a failed node, or restore the desired replica count. It should return a replacement to traffic only after readiness checks pass.
It is not self-understanding. If a bad release crashes every replacement, automation may create an endless restart loop. If a dependency is overloaded, adding callers may increase pressure. Human alerts, rollout controls, backoff, and a stop condition remain important.
The autoscaling and cloud best-practices article explains capacity adaptation. Autoscaling and self-healing can support availability, but neither fixes incorrect code, unsafe state changes, or a shared dependency failure.
Failback Is the Second Risky Move
Server A is repaired, but Aman does not immediately send half the evening traffic back. The node may need current configuration, warmed caches, state reconciliation, and proof that the original cause is gone.
Read the timeline beyond the first green moment. Failure detection leads to failover and a reduced but usable service. Replacement restores capacity. Canary traffic tests the repaired path. Only then does a gradual failback return the system to its intended shape.
A safe failback prevents flapping, where traffic repeatedly moves between unstable targets. Use a stability window, gradual ramp, monitoring, and a quick reversal path. For stateful systems, confirm which copy is authoritative before reversing replication or accepting writes.
Redundant Copies Can Share One Failure
Three replicas in one zone can lose the same power or network path. Ten identical instances can receive the same bad release. Separate servers can share one database, secret, certificate, configuration, or control plane. This is a common-mode failure: one cause defeats several supposedly independent copies.
Fault isolation limits that blast radius by placing components behind boundaries that do not all share capacity, deployment, state, or network fate. A boundary helps only if the system can keep useful work outside it operating safely.
Redundancy reduces risk only when the copies do not share every important failure mode.
Map dependencies and fault domains, vary deployment batches, isolate capacity, and test the actual failure boundary. When the entire location is lost, the solution moves beyond local HA. Disaster Recovery in System Design: RTO, RPO and DR handles that larger scope.
Rebuild the Architecture From One Falling Server
Picture the incident as one short film. Server A falls. The load balancer has no alternate. SnackNow adds a genuinely independent path with N+1 headroom. Health checks decide whether each node is ready. Failover protects checkout while optional features degrade. Self-healing replaces capacity. Canary traffic proves the repair, and failback returns gradually.
Now contrast the choices: active-active uses several live paths but coordinates concurrent state; active-passive simplifies ownership but pays for waiting capacity and promotion time. HA permits a bounded interruption; fault tolerance hides the covered failure. Neither survives a common-mode failure it was not designed to isolate.
The spare is not the design. Detection, capacity, state, switching, verification, and independence make the spare usable.
In an interview or workplace review, begin with the critical path and failure scope. Identify the single point, choose the alternate pattern, explain the failover sequence, then name the capacity and consistency cost. That sequence is much easier to remember than a list of availability products.
Lock in the takeaway
Frequently asked questions
What is a single point of failure in system design?
A single point of failure is one component whose loss can make a required service or critical journey unavailable by itself. It can be a server, database, network path, credential, configuration source, or another shared dependency.
Why do two servers not automatically provide high availability?
Both servers must be healthy, reachable, state-safe, and able to carry the remaining load, while traffic routing and dependencies must also survive. Two replicas that share one fatal dependency or lack a working failover path can still fail together.
What is the difference between high availability and fault tolerance?
High availability minimises downtime and may include a brief interruption during detection and failover. Fault tolerance aims to continue the required function without user-visible interruption for explicitly defined failure classes.
What is active-active vs active-passive?
In active-active, multiple nodes handle live traffic and survivors absorb work after a failure. In active-passive, one primary handles traffic while a standby waits for promotion, usually simplifying active ownership but adding promotion time and reserved capacity cost.
What does N+1 redundancy mean?
N is the capacity needed for the expected workload. N+1 deploys one additional equivalent unit so the system can lose one unit and still retain the required capacity, assuming routing, state, and dependencies also remain healthy.
How do health checks participate in failover?
Health checks provide signals about liveness and readiness. After a configured threshold, routing or orchestration removes an unhealthy instance, directs traffic to healthy capacity, and may start a replacement. Checks must represent the critical path without overreacting to optional dependency failures.
What is the difference between failover and failback?
Failover transfers work to a healthy alternative after failure. Failback returns work to the restored original or normal topology, ideally after state resynchronisation, readiness checks, canary traffic, and safeguards against repeated switching.
What is a common-mode failure?
A common-mode failure is one cause that defeats multiple redundant components together, such as a bad deployment, shared database, expired credential, single availability zone, common network path, or faulty configuration.
