Reliable, Scalable, Maintainable Applications
Designing Data-Intensive Applications · Chapter 1 · Recall lab
Chapter 1 doesn’t teach you a technology. It hands you three words to argue with, and insists that none of them is a badge you can pin on a system. Each is a question you keep re-asking as load, faults, and requirements move. Work the widgets below; the numbers are the point.
Data-intensive, not compute-intensive
The premise: for most applications today, raw CPU is rarely the binding constraint. What bites is the amount of data, its complexity, and the speed at which it changes. That’s what makes an application data-intensive.
The second premise is subtler and it’s the one people forget. The standard building blocks, databases, caches, search indexes, stream processors, batch processors, look like distinct, well-understood categories, but the boundaries have blurred. Message queues that offer durability look database-ish. Datastores used as queues. Redis. Kafka. So the moment you wire several of these together to meet requirements no single tool satisfies, you have designed a data system, and you own its guarantees, not the vendors.
Compose a system, click the parts you'd add
A “special-purpose data system from smaller, general-purpose components.” Each addition buys you something and hands you a problem.
Nothing selected yet. Start with the database.
0 component — you now own these guarantees- OLTP database — System of record. Now: what is the source of truth, and who is allowed to write to it?
- Cache — Reads get cheap, and you inherit invalidation. When the database changes, who expires the cache, and what does a client see in between?
- Search index — Full-text queries become possible, and now two stores must agree. The index lags the database; is that lag acceptable to the user?
- Message queue — Writes become asynchronous, so failure moves. If the consumer dies after the producer got its ack, is the work lost or retried, and is retrying safe?
- Batch / stream processor — Derived data appears. Now you own a recomputation story: if the logic was wrong last week, can you rebuild the outputs?
No single tool answers all of this. The code that stitches them together is the data system, and its guarantees are yours to define.
The chapter then narrows to the three concerns that dominate the rest of the book. Note the framing it uses at the end: reliability, scalability and maintainability are non-functional requirements, as opposed to functional requirements (what the system should do). They are not features you finish.
Continuing to work correctly, even when things go wrong
Unpack “working correctly” and you get four expectations: it does what the user expected; it tolerates the user making mistakes or using it in unexpected ways; performance is good enough at the expected load and data volume; and it prevents unauthorized access and abuse.
Fault
One component deviating from its spec. A disk returns garbage. A node stops responding. You cannot drive the probability of faults to zero.
Failure
The system as a whole stops providing the service the user needs. This is what you actually get to design against.
The whole discipline sits in the gap between those two boxes: build fault-tolerance mechanisms that stop faults from becoming failures, reliable systems out of unreliable parts. And counterintuitively, once you’ve built such machinery, it can make sense to raise the fault rate on purpose, kill processes at random, because a lot of critical bugs are really bad error handling, and untested recovery code is not recovery code. That’s the Chaos Monkey argument.
One exception to “prefer tolerating over preventing”: security. If an attacker has already exfiltrated the data, there is no cure to apply. Prevention is the only move.
Fault console, three families of things going wrong
Pick a fault. The console tells you which family it belongs to, whether it hits nodes independently or all at once, and what actually helps.
Correlation: Independent across machines
What helps: Component redundancy, RAID, hot-swap. At fleet scale, expect one per day per 10,000 disks and design software to survive losing the whole machine.
Correlation: Independent, with weak correlations (rack temperature)
What helps: ECC and redundancy. Weak correlation matters: a shared cause like heat can cluster otherwise independent failures.
Correlation: Independent, but blast radius depends on topology
What helps: Redundant paths, plus labelling and interfaces that make the wrong action hard.
Correlation: Independent, and routine
What helps: Some platforms deliberately favour flexibility and elasticity over single-machine reliability. Tolerate machine loss in software; use rolling upgrades to patch without downtime.
Correlation: CORRELATED, every node at once
What helps: No quick fix. Interrogate assumptions about the environment, test thoroughly, isolate processes, let them crash and restart, and measure behaviour in production. Redundancy does not help here: all copies run the same bug.
Correlation: Correlated wherever the code runs
What helps: Process isolation and resource limits, plus monitoring that catches the trend before exhaustion.
Correlation: Correlated across every caller
What helps: Timeouts, backpressure, and self-checking: if the system promises a guarantee, messages in equal messages out, have it check itself continuously and raise an alert on discrepancy.
Correlation: Correlated by construction
What helps: A small fault in one component triggers a fault in the next, which triggers more. Isolate, shed load, and think hard about the interactions between components.
Correlation: Whatever the config touches
What helps: Minimise the chance to err, sandbox the risky work, test at all levels, make rollback fast, and monitor in detail.
Correlation: Everything
What helps: Gradual rollout so an unexpected bug reaches a small subset of users, plus fast rollback and tools to recompute affected data.
Correlation: Irreversible
What helps: Here prevention beats cure, because no cure exists. You cannot un-leak data, so the fault-tolerance framing does not apply.
Hardware faults
Random and mostly independent. Answer with redundancy, RAID, dual PSUs, generators, and, as fleets grow, software tolerance of losing whole machines. Bonus: rolling upgrades instead of planned downtime.
Software errors
Systematic and correlated across nodes, so they cause far more failures than hardware does. They lie dormant until an assumption about the environment quietly stops being true.
Human errors
The leading cause of outages in one study of large internet services, operator configuration errors, with hardware implicated in only 10–25%. Humans are part of the system.
Disk arithmetic, why scale changes the conversation
Commodity disks are quoted at roughly 10–50 years MTTF. The individual number sounds reassuring. Multiply it by a fleet and it stops sounding reassuring.
2.74 disk deaths per day, expected
8.8 hours between failures
Designing around unreliable humans, five levers
- Minimize the chance to err. Abstractions, APIs and admin interfaces that make the right thing easy. But too restrictive and people route around them, which cancels the benefit.
- Decouple where mistakes happen from where they hurt. Full-featured sandboxes with real data and no real users.
- Test at every level, unit through whole-system integration, plus manual. Automation is especially good at the corner cases production rarely reaches.
- Make recovery fast. Quick config rollback, gradual code rollout so a bug hits few users, tools to recompute derived data.
- Monitor in detail. Performance metrics, error rates, telemetry, in the rocketry sense: early warning, plus the diagnostic record when something does go wrong.
How important is reliability?
The argument isn’t restricted to reactors and air traffic control. The chapter’s image is a parent whose only copies of their children’s photos live in your application. You may choose to trade reliability away, unproven market, thin margins, throwaway prototype, but the point is to know you’re doing it, and not to discover it later.
Coping with increased load
First, the ground rule the chapter is emphatic about: this is not a one-dimensional label. “X is scalable” and “Y doesn’t scale” are both meaningless sentences. The useful form is: if the system grows in this particular way, what are our options? Which means you need to say how it might grow, and that requires numbers.
Step 1 · Describing load
Load parameters are the handful of numbers that summarise current load. Which numbers depends entirely on your architecture: requests per second, read/write ratio, simultaneously active users, cache hit rate. Sometimes the average is what matters; sometimes a small number of extreme cases dominates the bottleneck.
The Twitter fan-out lab (figures as of Nov 2012)
Two operations. Posting a tweet ran ~4.6k requests/sec average, over 12k at peak. Reading a home timeline ran ~300k requests/sec. Handling 12k writes/sec is easy. The scaling problem isn’t tweet volume, it’s fan-out: each user follows many, and is followed by many.
345,000 writes/sec, one row per tweet writes/sec into timeline caches writes/sec for ordinary users
300,000 timeline reads/sec, each a join+merge cheap timeline lookups/sec reads/sec, mostly precomputed
Twitter started on approach 1, couldn’t keep up with timeline reads, and moved to approach 2, because publish rate is nearly two orders of magnitude below read rate, so it pays to do more work at write time. The cost: a single celebrity tweet can mean tens of millions of timeline writes, and the target is delivery within about five seconds. Hence the final twist, a hybrid, where celebrity tweets are exempted from fan-out and merged in at read time. The real load parameter here was never “tweets per second”; it was the distribution of followers per user.
Step 2 · Describing performance
With load described, you can ask the two growth questions: increase a load parameter and hold resources fixed, what happens to performance? Or: increase a load parameter and hold performance fixed, how much more resource do you need?
Batch systems, throughput
Records per second, or wall-clock time for a job over a dataset. Ideally dataset ÷ throughput; in practice longer, because of skew, data spread unevenly across workers, and waiting on the slowest task.
Online systems, response time
Time from the client sending a request to receiving a response. Not the same as latency: response time is what the client sees, including network and queueing delays; latency is the duration a request spends latent, waiting to be handled.
The signature instrument, a distribution, not a number
Repeat the identical request and you still get different response times: a context switch, a lost packet and TCP retransmission, a GC pause, a page fault, even mechanical vibration in the rack. So treat response time as a distribution of measured values. Each bar below is one request out of a hundred; height is how long it took.
Why the mean is a poor answer to “what’s typical”: it doesn’t tell you how many users actually experienced that delay. The median (p50) does, half of requests come back faster, half slower. Then higher percentiles size up the outliers: p95, p99,
p999 are the thresholds under which 95%, 99% and 99.9% of requests complete. These are your tail latencies.
Amazon specifies internal services at p999, 1 request in 1,000, for a reason that inverts the intuition: the slowest requests often belong to the accounts with the most data, i.e. the customers who have bought the most. The tail is disproportionately your best users. Supporting numbers from the chapter: 100 ms extra response time cost about 1% of sales; a 1-second slowdown cut a customer satisfaction metric by 16%. And a limit: optimizing p9999 was judged not worth it, at that level you’re fighting random events outside your control for diminishing returns.
Percentiles are also the currency of SLOs and SLAs: e.g. “up” defined as median under 200 ms and p99 under 1 s, available 99.9% of the time, with a refund attached when it isn’t met.
Queueing, head-of-line blocking, and where you measure
At high percentiles, much of the response time is queueing delay. A server processes only a few things in parallel, CPU cores, so a handful of slow requests can hold up everything behind them. Those later requests are fast to process; the client still sees them as slow. This is head-of-line blocking, and it’s the reason to measure response times client-side.
Corollary for load testing: the load generator must keep sending requests independently of response time. If it waits for each response before sending the next, it artificially shortens the queues and your measurements flatter you.
Tail latency amplification
Now the effect that makes tail percentiles matter far more than they look. If serving one end-user request requires several backend calls, even issued in parallel, the user waits for the slowest one. It takes a single slow call to make the whole request slow, so as the number of calls grows, the share of end-user requests that are slow climbs well above the per-call rate.
1 − (1 − p)n. A p99 backend and ten fan-out calls means roughly one in ten user requests lands in the tail.
9.56% of end-user requests hit the tail
Practical footnote on measuring percentiles: keep a rolling window, and if sorting every window is too expensive use an approximation, forward decay, t-digest, HdrHistogram. And never average percentiles across machines or time buckets; that is mathematically meaningless. Add the histograms instead.
Step 3 · Coping with load
An architecture fit for one level of load is unlikely to survive ten times that, so on a fast-growing service expect to rethink it at every order of magnitude, or more often.
Scaling up · vertical
A bigger machine. Simpler, often the right answer, and the historical default for databases, but high-end machines get expensive fast.
Scaling out · horizontal
Spread load over many smaller machines: a shared-nothing architecture. Very intensive workloads usually can’t avoid it. Good architectures are a pragmatic mixture, a few fairly powerful machines can beat a swarm of small VMs on both cost and sanity.
Elastic vs manual
Elastic systems add resources automatically when they detect load, useful when load is unpredictable. Manually scaled systems are simpler and hold fewer operational surprises.
Stateless vs stateful
Distributing stateless services is fairly straightforward. Taking a stateful data system from one node to many introduces a lot of complexity, hence the old wisdom: keep the database on one node until cost or availability forces your hand.
No magic scaling sauce
There is no generic, one-size-fits-all scalable architecture. A scalable architecture is built around assumptions about which operations are common and which are rare, that is, around your load parameters. Get those assumptions wrong and the scaling effort is wasted at best, counterproductive at worst. In an early-stage product, iterating on features usually beats scaling for hypothetical load.
The chapter’s clinching example: identical data throughput, entirely different systems.
| System | Request rate | Payload | Throughput |
|---|---|---|---|
| High-volume small requests | 100,000 / sec | 1 kB | 100 MB/s |
| Low-volume huge requests | 3 / min | 2 GB | 100 MB/s |
Most of the cost is after you ship
The majority of software cost isn’t initial development, it’s the ongoing work: fixing bugs, keeping things running, investigating failures, porting to new platforms, bending the system to new use cases, repaying technical debt, adding features. And much of that work is unloved, on systems people call legacy. Every legacy system is unpleasant in its own way, so the chapter’s move is preventative: design so that you don’t create one.
Operability
Make it easy for operations to keep the system running smoothly.
Simplicity
Make it easy for a new engineer to understand the system, by removing as much complexity as possible. Not the same as a simple user interface.
Evolvability
Make it easy to change the system later, for use cases nobody anticipated. Also called extensibility, modifiability, or plasticity.
Operability, making life easy for operations
The line worth memorising: good operations can often work around bad or incomplete software, but good software cannot run reliably with bad operations. Automation helps, but a human still has to build the automation and check that it works.
What good software gives operations, tick what your current system actually does
0 of 7 ticked. That's the full list of what good software does for its operations team. Note that every item is about making routine tasks easy, which frees the team for high-value work.The gaps are where operational effort quietly leaks. Each unticked item is a routine task someone is doing by hand.Worth noticing how much of maintainability is decided by design choices made long before handover.
And what the operations team is on the hook for
- Monitoring system health and restoring service quickly when it degrades
- Tracking down causes, failures, degraded performance
- Keeping software and platforms current, including security patches
- Watching how systems affect each other, so a bad change is caught before it lands
- Anticipating and pre-empting problems, e.g. capacity planning
- Establishing good practice and tooling for deployment, configuration management and more
- Complex maintenance work such as platform migrations
- Maintaining security as configuration changes accumulate
- Defining processes that make operations predictable, and keeping the environment stable
- Preserving organizational knowledge about the system even as individuals come and go
Simplicity, managing complexity
Complexity slows everyone down and raises the cost of change. The symptoms are recognisable: explosion of the state space, tight coupling of modules, tangled dependencies, inconsistent naming and terminology, hacks aimed at performance problems, special-casing to work around issues elsewhere. The name for the result is a big ball of mud. The concrete danger is not aesthetic: complexity hides bugs, makes hidden assumptions and unintended consequences easier to introduce, and makes estimates unreliable.
The key distinction is accidental complexity, complexity that is not inherent in the problem the software solves, but arises only from the implementation. And the best tool against it is a good abstraction: it hides implementation detail behind a clean, understandable façade, and it can be reused. High-level languages hide machine code and CPU registers; SQL hides on-disk structures and concurrent requests. Finding good abstractions for distributed systems is much harder, which is a preview of the rest of the book.
Essential or accidental?, click each to classify
The business genuinely bills in 14 currencies with different rounding rules
Essential. The domain really is this shape; no implementation choice makes it go away. Your job is to model it clearly, not to wish it smaller.
Three services each keep their own copy of the currency table, kept in sync by a nightly script
Accidental. Nothing in the problem requires three copies. This is tangled dependencies plus a hack, and it will drift.
Some modules call it customer, others account, others user_v2
Accidental. Inconsistent naming and terminology is a classic symptom, it forces every reader to hold a translation table in their head.
Regulation requires seven-year retention of every invoice version
Essential. An external requirement is part of the problem. You can implement it well or badly, but you cannot remove it.
A feature flag from two years ago is still branching in nine places
Accidental. State-space explosion: nine branches means far more reachable states than the behaviour actually needs.
A denormalized column added to make one dashboard fast, now written by four code paths
Accidental. A hack aimed at a performance problem, and now special-cased maintenance everywhere. Textbook big-ball-of-mud material.
Payments can fail asynchronously, hours after the user leaves the page
Essential. Asynchrony is in the domain here. A good abstraction can hide the mechanics; it cannot make the uncertainty disappear.
Deploying service A requires deploying service B in the same window
Accidental. Tight coupling of modules, the deploy order is telling you the boundary is in the wrong place.
0 of 0 classified correctly. Keep going, the pattern is that accidental complexity always has an implementation story attached, never a domain one. The distinction is the whole point of the section: accidental complexity is complexity that is not inherent in the problem the software solves, it arises only from the implementation. That's the part you can remove, and the tool for removing it is a good abstraction.
Evolvability, making change easy
Requirements will move: you learn new facts, unanticipated use cases appear, business priorities shift, the platform changes underneath you, legal or regulatory requirements arrive. Agile working patterns are the organisational answer, and TDD and refactoring are useful tools, but Agile’s techniques mostly address small, local scale. This chapter is after agility at the level of a whole data system: how do you refactor an architecture from one approach to another? The chapter’s own example is exactly the Twitter migration you played with above.
And the punchline that ties the pillar together: ease of modification is closely tied to simplicity and good abstractions. Simple, well-understood systems are usually easier to change than complicated ones, which is why the chapter uses evolvability for maintainability at the data-system level rather than treating it as a separate virtue.
Did it stick?
1. A single node's disk starts returning corrupted blocks, but the cluster serves every read correctly from replicas. What has occurred?
2. Which class of fault tends to cause more system failures, and why?
3. Why does the chapter call “our system is scalable” a meaningless statement?
4. Twitter moved from fan-out on read to fan-out on write. What made that the right trade?
5. For Twitter's fan-out design, which load parameter matters most?
6. Response time and latency, what's the distinction the chapter draws?
7. Why does Amazon specify internal services at p999 rather than the mean, but decline to optimize p9999?
8. Ten backend calls fan out to serve one user request, each with a 1% chance of being slow. Roughly what share of user requests are slow?
9. You're load-testing and your client waits for each response before sending the next request. What's wrong?
10. Which of these is accidental rather than essential complexity?
Answered 0 of 10 · 0 correct
The twenty terms the chapter actually installs
Chapter 1 in one breath. Reliability is making the system keep working correctly in the face of faults, hardware, software, human, by preventing faults from becoming failures. Scalability is having options when load grows, which first requires describing load with the right parameters and performance with percentiles rather than means. Maintainability is designing for the operators and engineers who come next: operability, simplicity, evolvability. All three are non-functional requirements, none has an easy fix, and the recurring patterns and tools for them are what the remaining twelve chapters are about.
Study aid built for recall practice. Concepts, figures and examples are Martin Kleppmann’s, from Designing Data-Intensive Applications, Chapter 1, go back to the book itself for his wording and the full reference list.