06 — Backend Performance: Asynchronism & Scaling (Don't Block The Request, Spread The Load)

Two roadmap sections that each fit on one page — Asynchronism (2 items) and Load Balancing & Scaling (2 items) [1] — turn out to be the two halves of the same move: get the long work off the request path so requests stay fast, and grow the boxes serving requests so more requests fit. I keep them together here.

Asynchronism — get the long work off the request path

Offload heavy tasks to background jobs or queues — long work done during a request is the worst-case for latency: the user waits for it to complete before their response starts. The two patterns that move it off the request path depend on whether the user needs the result. Fire-and-forget: the user requests "send me a receipt email," you enqueue a job and return 200 immediately; the worker sends the mail seconds later. Polling or WebSocket: the user requests "generate this report," you enqueue a job, return a 202 Accepted with a job id, and the client polls (or receives via WS) for completion. The HTTP request now returns in milliseconds regardless of how long the work takes. Candidates for moving off the request: image/media processing, PDF generation, third-party webhooks (don't make _their_ latency yours), batch operations over many rows.
Utilize message brokers for asynchronous communication between services — the cross-service version of the same idea. When service A calls service B synchronously, A's latency is at least B's latency, and B's outage is A's outage. A message broker (Kafka, RabbitMQ, NATS, SQS) decouples them: A publishes an event, returns; B consumes when it can. The wins: A is no longer blocked on B, B can absorb spikes (the queue grows, B catches up), and a B outage doesn't fail A's requests — it grows the queue. The cost: an eventually-consistent system (the two sides may briefly disagree) instead of a request-response one, so right for things that don't need to be synchronous (notifications, audit writes, "tell service C the user signed up") and wrong for things that do (the user's profile in the rendered response). The model: synchronous when the caller needs the result now; broker-mediated when "now" can be a few seconds or minutes.

Scaling & Load Balancing — give the requests more boxes

Use Horizontal or Vertical scaling whatever appropriate — the choice between bigger boxes (vertical: more CPU/RAM on one machine) and more boxes (horizontal: more machines). Vertical is simpler — no distributed state, no sharding — and right up to the ceiling of one machine, which is much higher than people assume. Horizontal scales past that ceiling, and buys redundancy (lose one box, others stay up), at the cost of state coordination. Rule of thumb I use: scale vertical until you can't (you're hitting one machine's CPU/RAM ceiling, or the cost of one bigger machine exceeds two smaller ones), then scale horizontal. Horizontal first is usually premature; horizontal-only is usually over-engineered.
Use load balancing to distribute traffic across servers — the companion to horizontal scaling. Multiple boxes are useless if traffic piles onto one; a load balancer (NGINX, HAProxy, AWS ALB, a service mesh's sidecar) spreads it. The balancer's choices are: algorithm (round-robin, least-connections, IP-hash for sticky sessions — round-robin is usually right, least-connections when requests vary in cost), health checks (so a sick box is removed automatically), and session persistence (so a user stays on one box when needed). The hidden win is the cost-cut from health checks — a box with degraded latency is removed from rotation automatically, instead of dragging down p99 for users routed to it.

Why these four live together

The two sections read separately because they're different mechanisms — a queue is asynchronous plumbing, a load balancer is synchronous plumbing — but they're doing the same job from opposite directions: stop letting slow work slow the fast path, stop letting one box bottleneck the fleet. A backend that ships both halves is a backend where requests are uniformly fast under load, because the long work is in a queue and the requests themselves are spread across boxes.

The single highest-leverage move of the four is usually offload heavy tasks to a queue: it converts a 2-second request into a 50ms request with no infra change, just an async refactor. The second is load balancing: it converts a one-box system into a multi-box system that survives single-node loss and absorbs spikes. Scaling decisions (horizontal vs vertical) follows from the workload, and the broker is usually the right architecture once you have more than one service that calls another. Four items, two halves, one shape: keep fast work fast, spread the load, and the rest of the system performs.

References

[1] roadmap.sh, "Backend Performance Best Practices — Asynchronism / Load Balancing & Scaling," roadmap.sh, 2024. [Online]. Available: https://roadmap.sh/backend-performance-best-practices
[2] RabbitMQ, "Work Queues," RabbitMQ Tutorial, 2024. [Online]. Available: https://www.rabbitmq.com/tutorials/tutorial-two-python
[3] NGINX, "Load Balancing," NGINX Docs, 2024. [Online]. Available: https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
[4] AWS, "Horizontal vs Vertical Scaling," AWS Reference Architecture, 2024. [Online]. Available: https://aws.amazon.com/compare/the-difference-between-vertical-and-horizontal-scaling/

Asynchronism — get the long work off the request path

Offload heavy tasks to background jobs or queues — long work done during a request is the worst-case for latency: the user waits for it to complete before their response starts. The two patterns that move it off the request path depend on whether the user needs the result. Fire-and-forget: the user requests "send me a receipt email," you enqueue a job and return 200 immediately; the worker sends the mail seconds later. Polling or WebSocket: the user requests "generate this report," you enqueue a job, return a 202 Accepted with a job id, and the client polls (or receives via WS) for completion. The HTTP request now returns in milliseconds regardless of how long the work takes. Candidates for moving off the request: image/media processing, PDF generation, third-party webhooks (don't make _their_ latency yours), batch operations over many rows.

Utilize message brokers for asynchronous communication between services — the cross-service version of the same idea. When service A calls service B synchronously, A's latency is at least B's latency, and B's outage is A's outage. A message broker (Kafka, RabbitMQ, NATS, SQS) decouples them: A publishes an event, returns; B consumes when it can. The wins: A is no longer blocked on B, B can absorb spikes (the queue grows, B catches up), and a B outage doesn't fail A's requests — it grows the queue. The cost: an eventually-consistent system (the two sides may briefly disagree) instead of a request-response one, so right for things that don't need to be synchronous (notifications, audit writes, "tell service C the user signed up") and wrong for things that do (the user's profile in the rendered response). The model: synchronous when the caller needs the result now; broker-mediated when "now" can be a few seconds or minutes.

Scaling & Load Balancing — give the requests more boxes

Use Horizontal or Vertical scaling whatever appropriate — the choice between bigger boxes (vertical: more CPU/RAM on one machine) and more boxes (horizontal: more machines). Vertical is simpler — no distributed state, no sharding — and right up to the ceiling of one machine, which is much higher than people assume. Horizontal scales past that ceiling, and buys redundancy (lose one box, others stay up), at the cost of state coordination. Rule of thumb I use: scale vertical until you can't (you're hitting one machine's CPU/RAM ceiling, or the cost of one bigger machine exceeds two smaller ones), then scale horizontal. Horizontal first is usually premature; horizontal-only is usually over-engineered.

Use load balancing to distribute traffic across servers — the companion to horizontal scaling. Multiple boxes are useless if traffic piles onto one; a load balancer (NGINX, HAProxy, AWS ALB, a service mesh's sidecar) spreads it. The balancer's choices are: algorithm (round-robin, least-connections, IP-hash for sticky sessions — round-robin is usually right, least-connections when requests vary in cost), health checks (so a sick box is removed automatically), and session persistence (so a user stays on one box when needed). The hidden win is the cost-cut from health checks — a box with degraded latency is removed from rotation automatically, instead of dragging down p99 for users routed to it.

Why these four live together

06 — Backend Performance: Asynchronism & Scaling (Don't Block The Request, Spread The Load)

Asynchronism — get the long work off the request path

Scaling & Load Balancing — give the requests more boxes

Why these four live together

References

Comments