02 — Performance vs Scalability, and the Latency/Throughput Pair

"The system is slow" used to mean one thing to me, and it was two different problems wearing the same sentence. Writing them down separated them cleanly: a performance problem is about a single user; a scalability problem is about behavior under load. [1] And the two dials you turn to chase either one are latency (how long one request takes) and throughput (how many requests you serve at once) [2][3].

The framing that finally landed for me is treating the two pairs as orthogonal axes. Performance vs scalability is a _diagnosis_ question — which kind of problem do I have. Latency vs throughput is a _measurement_ question — which numbers describe the system. Once I stopped mixing them, "make it fast" turned into two precise questions: fast for whom, and fast for how many at once.

Performance vs scalability: diagnosis

The roadmap's one-line test is the part worth memorizing [1]:

• If you have a performance problem, your system is slow for a single user.
• If you have a scalability problem, your system is fast for a single user but slow under heavy load.

A service is scalable if adding resources yields a proportional increase in performance — serving more units of work, or handling larger units of work as datasets grow [1]. The diagnostic habit is now my first move: I reproduce the slowness with one user. If it's slow even alone, that's performance, and throwing more servers at it will not help — I need to fix the single-request path (a slow query, a blocking call, an inefficient algorithm). If it's fast alone but degrades as concurrency climbs, that's scalability, and the fix is about spreading or buffering the load.

Latency vs throughput: the two dials

Once I know which problem I have, I describe it with two numbers [2][3]:

• Latency — the time it takes for the system to respond to a single request.
• Throughput — the number of requests the system can handle at the same time.

They pull against each other. A service that handles one request at a time has low throughput and predictable latency. A service that fans out to thousands of concurrent workers can have huge throughput, but each request may experience higher latency from contention. The guidance the roadmap repeats is the one I now aim for: maximal throughput with acceptable latency [2] — and "acceptable" is a product decision, not a technical one. A video stream tolerates seconds of latency; a chat app does not.

A useful mental hook is Little's Law, which links the two through concurrency: the average number of items in a system equals the average arrival rate multiplied by the average time each spends in the system [4]. It tells me that if I want to hold latency constant while doubling throughput, I have to roughly double the in-flight capacity — more workers, more connections, more parallelism.

How I use this

The two-step habit is the whole payoff. First, reproduce the slowness with one user to decide performance vs scalability — this stops me from "fixing" a scalability problem by optimizing a single query that was never the bottleneck. Second, name the target in latency-and-throughput terms before changing anything, because "make it fast" means different things to a batch pipeline (maximize throughput) and a user-facing endpoint (minimize p99 latency — the slowest response almost every user sees). Without that target, I have no way to know whether the change worked.

References

[1] Professor Beekums, "Performance vs scalability," blog.professorbeekums.com, 2017. [Online]. Available: https://blog.professorbeekums.com/performance-vs-scalability/

[2] cs.fyi, "System design: Latency vs throughput," 2021. [Online]. Available: https://cs.fyi/guide/latency-vs-throughput/

[3] Cadence, "Understanding latency versus throughput," Cadence Blogs, 2022. [Online]. Available: https://community.cadence.com/cadence_blogs_8/b/fv/posts/understanding-latency-vs-throughput

[4] "Little's law," Wikipedia. [Online]. Available: https://en.wikipedia.org/wiki/Little%27s_law

[5] J. Bonér, "Scalability, availability & stability patterns," SlideShare, 2014. [Online]. Available: https://www.slideshare.net/jboner/scalability-availability-stability-patterns/

[6] W. Vogels, "A word on scalability," All Things Distributed, 2006. [Online]. Available: https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html