--- title: "08 — Backend Performance: Observability & Performance Testing (See, Then Prove)" uid: backend-performance-observability-testing tags: ["observability", "logging", "grafana", "monitoring", "prometheus", "roadmap:backend-performance", "backend", "load-testing", "performance"] excerpt: "You can't optimize what you can't see, and you can't keep it optimized if you don't keep testing it — the two smallest roadmap sections carry the whole discipline." date: 2026-08-13T04:50:26+0000 source: https://www.aveshina.my.id/en/blog/backend-performance-observability-testing --- The smallest sections close the roadmap — **Monitoring and Logging** (3 items) and **Performance Testing** (1 item) [1] — and they're the same move from two directions: **see what the system is doing, and prove it stays fast under load.** Four items total, one idea: no backend optimization survives without these. ## Monitoring and Logging — see what the system is actually doing - **Implement comprehensive monitoring and logging to track performance metrics and troubleshoot issues** — the comprehensive item is comprehensive on purpose: it's not one signal, it's a stack. Three layers I keep separate: - **Metrics** — aggregate numbers, sampled at intervals. Requests per second, p50/p95/p99 latency (the time that half, 95%, and 99% of requests stay under), error rate, queue depth. The dashboard view: how is the system doing _right now_? Built for trend-watching and alerting. - **Logs** — per-event records, structured (JSON), queryable. The post-mortem view: why did _this_ request fail? Built for hunting. - **Traces** — distributed traces that follow one request across services. The architecture view: where did the 500ms go? Built for attribution. - The mistake is to ship one (usually logs) and call observability done; you will find out fast that you have logs and no alerting on the latency nobody noticed creep. - **Use tools like Prometheus, Grafana, ELK stack** — the standardization of those three layers. Prometheus (metrics), Grafana (dashboards on top of Prometheus, Loki, Jaeger), the ELK or Loki/Mimir stack for logs. The choice matters less than the consistency: pick one stack per layer and use it everywhere. A service that uses Datadog while a sibling uses Prometheus means a person debugging across both has two ways of thinking and two query languages, and that cost is paid in latency-to-deploy-fixes. - **Use asynchronous logging mechanisms to minimise the logging overhead** — the trap of observability is that it slows the thing being observed. Synchronous logging writes the log line in-band with the request handler; a log that hits disk on every request adds milliseconds to every request, and when the disk is slow, latency collapses. Asynchronous logging (a queue with a writer thread) decouples the write from the request, so the request returns before the disk write happens. Most logging libraries have an async variant (loguru.enqueue, SLF4J's async appender, Python QueueHandler); the default is often sync. Turn it on in production; the caveat is queue-backpressure at very high log rates, which is itself a signal you're logging too much. ## Performance Testing — prove it stays fast under load - **Conduct regular performance testing and benchmarking to identify performance regressions, track improvements, and fine-tune optimization efforts over time** — the single item that closes the roadmap. Three reasons regular load testing matters: - **Regression detection** — the slow drift is invisible without a load test; "p99 went from 200ms to 500ms over six months and nobody noticed" is the failure mode every team eventually hits. An automated load test in CI catches the change between builds. - **Improvement validation** — every optimization in the previous seven notes earns its claim only when proven under load; k6 or Locust or JMeter running a representative flow, comparing before/after, is the only proof that exists. - **Capacity planning** — you don't know what your system can take until you overload it deliberately. The questions you should be able to answer: at what request rate (RPS) does your slowest 1% of requests (the p99) cross the latency you promised (your SLA)? At what RPS do your worker pools exhaust? At what RPS does the database saturate? The rhythm that pays: a short scheduled load test in CI per commit (catch regressions), a longer soak test before releases (catch slow leaks), and a deliberate stress test before launches (find the cliff). ## Why these two sections belong together The first seven notes on the backend roadmap are **changes you make** — config, code, indexes, queues. The last two sections are **the discipline that lets you know whether the changes worked.** Without monitoring the changes you made don't have numbers; without load testing the numbers it works for are only the traffic you happen to have. The double mistake the roadmap guards against: optimizing without benchmarking first (you can't claim a fix without a baseline) and benchmarking without monitoring (you can't trust a benchmark result without knowing what else the system was doing). Together they're not optional polish — they're the loop that closes all the rest of the roadmap. Every other backend item on the list is a _change_; these four are the _measurement_ that earns the change the right to ship. Get these right and the rest of the roadmap is iterative; get them wrong and the rest is guessing. ## References - [1] roadmap.sh, "Backend Performance Best Practices — Monitoring and Logging / Performance Testing," roadmap.sh, 2024. [Online]. Available: [https://roadmap.sh/backend-performance-best-practices](https://roadmap.sh/backend-performance-best-practices) - [2] Prometheus, "Best Practices," Prometheus Docs, 2024. [Online]. Available: [https://prometheus.io/docs/practices/](https://prometheus.io/docs/practices/) - [3] Grafana, "Observability," Grafana Documentation, 2024. [Online]. Available: [https://grafana.com/docs/grafana/latest/observability/](https://grafana.com/docs/grafana/latest/observability/) - [4] k6, "Load Testing," k6 Documentation, 2024. [Online]. Available: [https://k6.io/docs/](https://k6.io/docs/)