The Promise an App Must Keep: System Reliability Explained Through SnackNow

At 7:07 PM, the App Is Online but the Order Is Wrong

Riya opens SnackNow during the evening Chai + Samosa offer. The menu loads. She adds two items, pays, and sees a success message. A few seconds later, the confirmation shows the wrong quantity. Another customer receives a duplicate confirmation. The homepage is still loading normally.

Meera looks at the green status page and asks, "But the app is online. How can this be a reliability incident?"

Aman points to the payment record and the two conflicting confirmations. "Because online only tells us that some part of the system answered. It does not prove that the system kept its promise."

That is the question this article will answer: can an available application still be unreliable?

Read the incident timeline from left to right. Failure begins at 7:07 PM, monitoring detects it at 7:13, the faulty path is isolated at 7:21, ordering is restored at 7:28, and the team verifies the result before declaring recovery. Those timestamps will later become reliability metrics.

Reliability Is the Promise, Not the Green Light

System reliability is the ability of a system to perform its intended function correctly and consistently, under stated conditions, over a meaningful period. The phrase intended function matters. SnackNow exists to create the right order, charge the right amount, preserve the result, and let the customer understand what happened.

A homepage that answers while checkout creates duplicates is available at the surface and unreliable in the user journey. A database that stores every wrong order durably is durable and still incorrect. A service that recovers quickly after a crash may be resilient, yet recurring crashes can still make it unreliable over time.

Reliability asks whether the complete user promise keeps working correctly over time, including when parts of the system fail.

This framing gives engineering a useful boundary. Instead of asking whether every server is green, Aman writes the promise as an observable outcome: one successful payment should lead to one correct, durable, confirmable order.

The next design question is whether that promise still has another healthy path when a component disappears. That narrower problem is explained in High Availability vs Fault Tolerance: Failover Explained.

Five Different Questions About the Same Order

The words availability, correctness, durability, resilience, and reliability often arrive together. They become simpler when all five are asked about one order instead of five unrelated examples.

The table is not a vocabulary test. Read the first column and picture Riya's order. Each concept protects a different part of the same experience.

The visual shows why these properties can disagree. SnackNow can answer requests but return the wrong result. It can save the wrong result perfectly. It can recover from a crash but repeat the same bug tomorrow. Reliability is the wider judgement across time.

Availability

Availability is the proportion of eligible time or requests in which a service is usable. Define the eligible population carefully. A health endpoint returning 200 is weak evidence if customers cannot complete checkout.

Correctness and durability

Correctness asks whether the result matches the rules. Idempotency, state validation, transaction boundaries, and reconciliation can help SnackNow avoid duplicate or impossible orders. Durability asks whether a successfully committed fact survives the failures covered by the storage design.

Replication may improve durability and availability, but live copies can reproduce a deletion or corrupt state. Historical recovery needs a different control. Backup and Recovery Strategies: RPO, RTO and PITR follows that exact problem.

Resilience and reliability

Resilience is the ability to withstand, contain, adapt to, and recover from disruption. Reliability is the broader observed result over time. A resilient system may degrade briefly and recover; a reliable service keeps the important customer promise within its stated target.

A reliable system is not one that never fails. It is one where failure does not become chaos.

The Incident Clock: MTBF, Detection, and MTTR

At 7:07 PM the checkout failure begins. At 7:13 monitoring notices an unusual gap between successful payments and confirmed orders. At 7:21 Aman isolates the faulty order worker. At 7:28 correct checkout is restored and verified.

Mean Time Between Failures is useful only after the team defines a failure. Is one wrong confirmation a failure, or only a complete outage? The answer changes the measurement. MTBF is usually most meaningful for repairable components or recurring incident classes, not as a universal score for an entire company.

MTTR is also ambiguous. Teams use the final R for repair, recovery, restore, or resolution. SnackNow must write down which interval it measures. Here it means time from customer-impacting failure to verified restoration of correct checkout.

A high MTBF is desirable because failures are less frequent. A low MTTD and MTTR are desirable because harm lasts for less time. Prevention, detection, containment, and recovery all matter. Pretending failure can be eliminated usually produces a system that is surprised when failure arrives.

The Nines Turn Reliability Into a Budget

An availability target converts vague ambition into an allowed failure budget. Over a 30-day month, 99% availability allows about 7 hours 12 minutes of unavailability. 99.9% allows about 43 minutes 12 seconds. 99.99% allows about 4 minutes 19 seconds.

Read the visual vertically. Each extra nine removes most of the remaining downtime budget. That usually demands stronger isolation, faster detection, safer deployments, more tested redundancy, and more operational effort.

Not every SnackNow feature needs the same target. Checkout and confirmed-order lookup carry direct money and trust risk. Reviews and recommendations can tolerate more interruption. One blanket target would either over-engineer low-impact features or under-protect the critical path.

Availability also needs a user-centred indicator. The fraction of successful checkout journeys is more useful than the fraction of healthy containers. A detailed system performance investigation can supply latency and error evidence, but reliability chooses which customer outcome that evidence must protect.

SLI, SLO, and SLA on One Promise Board

Aman writes three lines on the reliability promise board:

• SLI: the measured percentage of eligible checkouts that produce exactly one confirmed order after successful payment.

• SLO: the internal target, such as 99.95% successful checkouts in a calendar month.

• SLA: the external or contractual commitment, including scope and what happens if the agreed level is missed.

The SLI is evidence. The SLO is the engineering target. The SLA is a promise made outside the engineering room. They should describe the same customer journey, but they do not have to use identical thresholds. A stricter internal SLO can give the team room to act before an SLA is breached.

The key lesson from Google SRE guidance on service level objectives is to begin with behaviour users care about, then choose a useful indicator. Starting with an easy infrastructure metric can make a dashboard look precise while the customer promise remains invisible.

Why a Distributed Checkout Has More Ways to Fail

Riya's checkout crosses the frontend, API, order service, payment provider, database, network, and notification path. These components do not fail only as completely up or completely down. They can be slow, partitioned, stale, duplicated, partially successful, or unable to confirm what another component already did.

The visual should be read as a chain of promises. The payment provider may succeed while the order service times out. A retry can then create a duplicate unless the operation is idempotent. The database may be available while the queue is delayed. A notification can fail after the order is already correct.

Distributed reliability therefore needs explicit states, request identifiers, timeouts, bounded retries, idempotency, durable events, reconciliation, and end-to-end observability. The CAP theorem and storage fundamentals lesson explains why network partitions force data tradeoffs. The message queues lesson shows how asynchronous work can be decoupled without pretending delivery is magically exactly once.

A green component dashboard cannot prove a healthy business journey. Measure the journey and keep enough correlation evidence to reconstruct where its promise broke.

Improve Reliability Without Building a Space Programme

SnackNow does not need the most expensive architecture for every feature. It needs a deliberate response to its most damaging failures.

1. Write the critical promise. Define the customer outcome and its failure boundary.

2. Measure the journey. Track success, latency, duplication, stale state, detection time, and recovery time.

3. Remove dangerous single points. Add an alternate path only where the impact justifies its cost.

4. Make state transitions recoverable. Use idempotency, durable facts, reconciliation, and tested rollback or restore.

5. Practise failure. Test health checks, degraded modes, failover, recovery ownership, and communication.

Each control has a tradeoff. Redundancy adds cost and coordination. Retries add load and duplication risk. Durable writes can add latency. Monitoring adds storage and operational noise. The correct design spends complexity where it protects a valuable promise.

If a whole location disappears, local redundancy is no longer enough. Disaster Recovery in System Design: RTO, RPO and DR explains how the same promise is restored across a larger failure boundary.

Reliability engineering is the practice of choosing which failures matter, limiting their harm, and proving that recovery works.

Rebuild the Lesson From Five Objects

Close the article and picture five objects in order. The green status page asks whether the app is reachable. Riya's receipt asks whether the result is correct. The saved payment record asks whether the fact is durable. The incident clock measures detection and recovery. The promise board turns the customer journey into an SLI, SLO, and SLA.

Now retell the incident in one breath: SnackNow was available but produced an incorrect order, so the team defined reliability around the complete checkout promise. It separated availability, correctness, durability, and resilience; measured recurrence, detection, and recovery; set a user-centred objective; and improved the critical path without giving every feature the same expensive target.

A reliable system is not one that never fails. It is one where failure does not become chaos.

That is also a practical interview answer. Start with the intended function, show how availability can disagree with correctness, define the failure boundary and metrics, then explain the design choice and its cost. The incident does the remembering for you.