--- title: "02 — Performance vs Scalability, and the Latency/Throughput Pair" uid: performance-and-scalability tags: ["latency", "scalability", "roadmap:system-design", "throughput", "performance", "system-design"] excerpt: "Performance is about a single user; scalability is about behavior under load. Latency and throughput are the two dials you balance to chase either one." date: 2026-08-13T03:27:35+0000 source: https://www.aveshina.my.id/en/blog/performance-and-scalability --- "The system is slow" used to mean one thing to me, and it was two different problems wearing the same sentence. Writing them down separated them cleanly: **a performance problem is about a single user; a scalability problem is about behavior under load.** [1] And the two dials you turn to chase either one are **latency** (how long one request takes) and **throughput** (how many requests you serve at once) [2][3]. The framing that finally landed for me is treating the two pairs as orthogonal axes. Performance vs scalability is a _diagnosis_ question — which kind of problem do I have. Latency vs throughput is a _measurement_ question — which numbers describe the system. Once I stopped mixing them, "make it fast" turned into two precise questions: fast for whom, and fast for how many at once. ## Performance vs scalability: diagnosis The roadmap's one-line test is the part worth memorizing [1]: - If you have a **performance** problem, your system is slow for a single user. - If you have a **scalability** problem, your system is fast for a single user but slow under heavy load. A service is **scalable** if adding resources yields a proportional increase in performance — serving more units of work, or handling larger units of work as datasets grow [1]. The diagnostic habit is now my first move: I reproduce the slowness with one user. If it's slow even alone, that's performance, and throwing more servers at it will not help — I need to fix the single-request path (a slow query, a blocking call, an inefficient algorithm). If it's fast alone but degrades as concurrency climbs, that's scalability, and the fix is about spreading or buffering the load. ```figure ``` ## Latency vs throughput: the two dials Once I know which problem I have, I describe it with two numbers [2][3]: - **Latency** — the time it takes for the system to respond to a single request. - **Throughput** — the number of requests the system can handle at the same time. They pull against each other. A service that handles one request at a time has low throughput and predictable latency. A service that fans out to thousands of concurrent workers can have huge throughput, but each request may experience higher latency from contention. The guidance the roadmap repeats is the one I now aim for: **maximal throughput with acceptable latency** [2] — and "acceptable" is a product decision, not a technical one. A video stream tolerates seconds of latency; a chat app does not. A useful mental hook is Little's Law, which links the two through concurrency: the average number of items in a system equals the average arrival rate multiplied by the average time each spends in the system [4]. It tells me that if I want to hold latency constant while doubling throughput, I have to roughly double the in-flight capacity — more workers, more connections, more parallelism. ## How I use this The two-step habit is the whole payoff. First, reproduce the slowness with one user to decide performance vs scalability — this stops me from "fixing" a scalability problem by optimizing a single query that was never the bottleneck. Second, name the target in latency-and-throughput terms before changing anything, because "make it fast" means different things to a batch pipeline (maximize throughput) and a user-facing endpoint (minimize p99 latency — the slowest response almost every user sees). Without that target, I have no way to know whether the change worked. ## References [1] Professor Beekums, "Performance vs scalability," blog.professorbeekums.com, 2017. [Online]. Available: [https://blog.professorbeekums.com/performance-vs-scalability/](https://blog.professorbeekums.com/performance-vs-scalability/) [2] cs.fyi, "System design: Latency vs throughput," 2021. [Online]. Available: [https://cs.fyi/guide/latency-vs-throughput/](https://cs.fyi/guide/latency-vs-throughput/) [3] Cadence, "Understanding latency versus throughput," Cadence Blogs, 2022. [Online]. Available: [https://community.cadence.com/cadence_blogs_8/b/fv/posts/understanding-latency-vs-throughput](https://community.cadence.com/cadence_blogs_8/b/fv/posts/understanding-latency-vs-throughput) [4] "Little's law," Wikipedia. [Online]. Available: [https://en.wikipedia.org/wiki/Little%27s_law](https://en.wikipedia.org/wiki/Little%27s_law) [5] J. Bonér, "Scalability, availability & stability patterns," SlideShare, 2014. [Online]. Available: [https://www.slideshare.net/jboner/scalability-availability-stability-patterns/](https://www.slideshare.net/jboner/scalability-availability-stability-patterns/) [6] W. Vogels, "A word on scalability," All Things Distributed, 2006. [Online]. Available: [https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html](https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html) ```quiz Q: Your endpoint is fast for one user but slows down badly under heavy traffic. That is a ____ problem. - performance - scalability correct: 1 explain: Fast for one user, slow under load is the textbook scalability problem. Performance problems are slow even for a single user. Q: Latency measures ____, throughput measures ____. - time per request; requests handled at the same time - requests handled at the same time; time per request correct: 0 explain: Latency is the time to respond to a single request. Throughput is how many requests the system handles concurrently. Q: A service is "scalable" if… - adding resources yields a proportional increase in performance - it never experiences any downtime correct: 0 explain: Scalability is specifically about proportional performance gains as resources are added — more units of work handled, or larger units handled as data grows. Q: The usual target when tuning latency and throughput is… - minimal latency at any cost to throughput - maximal throughput with acceptable latency correct: 1 explain: The roadmap's repeated guidance is maximal throughput with acceptable latency. "Acceptable" is a product decision, not a universal constant. ```