--- title: "14 — Observability, Lifecycle & Compliance: Running APIs in Production" uid: observability-lifecycle-compliance tags: ["observability", "gdpr", "rabbitmq", "lifecycle", "compliance", "messaging", "pii", "kafka", "roadmap:api-design"] excerpt: "Three concerns separate a working API from a production-grade one: observability answers 'what's happening and why,' lifecycle answers 'how does this change safely over years,' and compliance answers 'what may this API touch.'" date: 2026-08-13T03:28:32+0000 source: https://www.aveshina.my.id/en/blog/observability-lifecycle-compliance --- An API can pass every test and still die in production — the difference is the operational layer above the code. The model that finally clicked: **these three are the operational layer above the code — observability answers "what's happening and why," lifecycle answers "how does this evolve without breaking anyone," and compliance answers "what is this API allowed to touch and under what rules."** [1][5][9] An API without them works in development and fails in production, silently. The thread connecting these roadmap sections is that they're all about *running* an API, not *building* it. You can ship an API with no observability, no version sunset plan, and no compliance review — and it will work, until something goes wrong and you can't explain why, or a regulator asks how you handle personal data, or a breaking change strands your customers. These practices are insurance against the failures that only show up at scale and over time. ## Observability: seeing inside a running API **Observability** is the ability to understand what's happening inside a running system from the data it produces — without reproducing the problem locally [1]. The three signals that define it: - **Logs** — discrete events ("request started," "payment failed for order 42"). Good for tracing specific incidents. - **Metrics** — aggregated numbers over time (request rate, error rate, latency percentiles, CPU). Good for spotting trends and setting alerts. - **Traces** — a request's path through every service it touches, with timing for each hop. Good for finding *where* the latency comes from. The questions observability lets you answer that you otherwise can't: "why did this specific request fail?", "where is the latency coming from — our code, the database, or a downstream service?", "is this API degrading slowly over the past hour?" [1] The discipline is to instrument every request with a **correlation ID** that ties the logs, metrics, and traces for that one request together, so an incident can be followed end to end. An observable API turns "the API is being weird" into "request abc-123 spent 4.2 seconds in the payments service, which returned a 503." ## The message brokers: Kafka and RabbitMQ The roadmap nests two specific messaging technologies under observability-adjacent concerns, and they're worth a focused note because they're the infrastructure that makes the asynchronous patterns from the previous post real. **Apache Kafka** is a distributed **event streaming** platform — a durable, partitioned, replayable log of events that many consumers can read independently [2]. Its model is fundamentally different from a simple queue: events are written to an append-only log and retained (for days or forever), so consumers can replay history, multiple consumers can read the same stream, and throughput scales horizontally. Kafka is the right choice for high-volume event streams, real-time data pipelines, and the backbone of an event-driven architecture at scale. **RabbitMQ** is a traditional **message broker** implementing AMQP (a standard protocol for message queues) — a queue-based system where messages are produced, delivered to a consumer, and acknowledged [3]. Its model is work-distribution: a message is consumed by exactly one worker, routing is flexible (topics, exchanges, routing keys), and messages are removed once acknowledged. RabbitMQ is the right choice for task queues, work distribution across workers, and request/reply patterns where each message should be handled once. ```figure Kafka — durable log, many readers, replay | e1 | e2 | e3 | e4 | e5 | (retained log) consumer A consumer B replay each consumer reads at its own offset; events retained RabbitMQ — queue, one consumer per message queue worker 1 worker 2 worker 3 → one message → exactly one worker, then removed ``` The mental shortcut: **Kafka for streams many services read and re-read; RabbitMQ for tasks handed to one worker.** Picking wrong is expensive — Kafka for simple task distribution is over-engineered, and RabbitMQ for a high-volume replayable event stream will fight you. ## API lifecycle management: design to retire **API lifecycle management** is the discipline of shepherding an API from initial design through deployment, evolution, and eventual retirement [4][5]. An API isn't built once; it lives for years, and the lifecycle is what keeps that long evolution safe. The stages I think in: 1. **Design** — model the resources, write the OpenAPI spec, review the contract before code. 2. **Build & test** — implement, with the testing pyramid from the previous post. 3. **Deploy** — versioned, documented, behind a gateway. 4. **Evolve** — additive changes ship freely; breaking changes go through a version bump with a documented migration path. 5. **Deprecate & retire** — when a version must die, communicate the sunset date, give consumers a generous window, monitor usage until it's near zero, then remove. The mistake lifecycle management exists to prevent: a breaking change shipped with no warning, or an old version yanked with no sunset window. Both strand consumers and destroy trust. The discipline is treating the API as a long-lived product with users who depend on stability, not as code you can refactor freely. A deprecation policy that's written down *before* you need it is the difference between a smooth version migration and an incident. ## Standards and compliance: the law is part of the contract **Standards and compliance** is the recognition that an API operates inside legal and regulatory constraints, and the contract isn't just "what fields come back" — it's "what data may this API touch, who may see it, and under what rules." [6] The specific regulations the roadmap calls out, and what each one actually demands of an API: - **GDPR** (General Data Protection Regulation, EU) — governs personal data of EU residents. APIs handling it need explicit consent, the right to erasure ("delete me"), data portability, and privacy-by-design [7]. A DELETE /users/me that doesn't actually delete downstream copies is a GDPR bug. - **CCPA** (California Consumer Privacy Act) — the California equivalent. APIs must support consumer requests for data access, deletion, and opt-out of sale [8]. - **PII** (Personally Identifiable Information) — the category of data these regulations protect: names, emails, addresses, IDs, anything that identifies a person [9]. The design rule is to collect the minimum PII necessary, protect it (encryption, access control), and know everywhere it flows. - **PCI DSS** (Payment Card Industry Data Security Standard) — applies to any API that processes, stores, or transmits card data [10]. The escape hatch most teams use: don't handle raw card numbers at all — pass them through a PCI-compliant payment provider (Stripe, Adyen) so your API never touches the sensitive data and stays out of PCI scope. - **HIPAA** (Health Insurance Portability and Accountability Act) — governs protected health information in the US. A healthcare API must encrypt PHI, enforce strict access controls, keep audit logs, and have business-associate agreements in place with any sub-processor [11]. The pattern across all of them: **know what regulated data your API touches, map everywhere it flows, and design the contract around the strictest rule that applies.** Compliance isn't a feature you bolt on at the end; it's a constraint on the design itself. An API that's been live for a year collecting PII with no retention policy and no deletion path is already a compliance problem, not a future one. ## How I use this For any production API, I treat three things as table stakes. **Observability**: every request gets a correlation ID, every service emits logs/metrics/traces, and the alerts trigger on symptoms a user would actually feel (error rate, latency) rather than raw resource usage. **Lifecycle**: a versioned contract, a written deprecation policy, and additive-by-default evolution so the API can change without breaking consumers. **Compliance**: an explicit map of what regulated data the API touches and where it flows, the strictest applicable rule applied as a design constraint (not a post-launch patch), and a deliberate choice to stay out of scope where possible (e.g., routing card data through a PCI-compliant provider). The unifying habit is treating the API as a long-lived, regulated, observable product from day one — not as code that becomes those things after it hurts. ## References [1] Shubhadeep Chattopadhyay, "Understanding API Observability," Medium, 2024. [Online]. Available: [https://medium.com/@shubhadeepchat/understanding-api-observability-cd0c61392dec](https://medium.com/@shubhadeepchat/understanding-api-observability-cd0c61392dec) [2] Apache Software Foundation, "Apache Kafka," 2024. [Online]. Available: [https://kafka.apache.org/](https://kafka.apache.org/) [3] Broadcom, "RabbitMQ," 2024. [Online]. Available: [https://www.rabbitmq.com/](https://www.rabbitmq.com/) [4] Postman, "What is the API Lifecycle?," 2024. [Online]. Available: [https://www.postman.com/api-platform/api-lifecycle/](https://www.postman.com/api-platform/api-lifecycle/) [5] Swagger, "What is API Lifecycle Management?," 2024. [Online]. Available: [https://swagger.io/blog/api-strategy/what-is-api-lifecycle-management/](https://swagger.io/blog/api-strategy/what-is-api-lifecycle-management/) [6] Traceable, "What is API Compliance and Why is it important?," 2024. [Online]. Available: [https://www.traceable.ai/blog-post/achieve-api-compliance](https://www.traceable.ai/blog-post/achieve-api-compliance) [7] GDPR, "General Data Protection Regulation," gdpr-info.eu. [Online]. Available: [https://gdpr-info.eu/](https://gdpr-info.eu/) [8] Cloudflare, "What is the CCPA?," 2024. [Online]. Available: [https://www.cloudflare.com/en-gb/learning/privacy/what-is-the-ccpa/](https://www.cloudflare.com/en-gb/learning/privacy/what-is-the-ccpa/) [9] IBM, "What is Personally Identifiable Information?," 2024. [Online]. Available: [https://www.ibm.com/topics/pii](https://www.ibm.com/topics/pii) [10] IT Governance, "What is PCI DSS and how to comply?," 2024. [Online]. Available: [https://www.itgovernance.co.uk/pci_dss](https://www.itgovernance.co.uk/pci_dss) [11] U.S. Department of Health & Human Services, "HIPAA," 2024. [Online]. Available: [https://www.hhs.gov/hipaa/index.html](https://www.hhs.gov/hipaa/index.html) ```quiz Q: The three signals of observability are… - logs, metrics, and traces - uptime, downtime, and restarts correct: 0 explain: Logs capture events, metrics capture aggregated numbers over time, and traces capture a request's path through services. Together they let you explain what a running API is doing. Q: What distinguishes Kafka from a simple message queue like RabbitMQ? - Kafka is a durable, retained log that many consumers can read and re-read independently; RabbitMQ delivers each message to one worker then removes it - Kafka only supports one consumer per topic correct: 0 explain: Kafka's log is append-only and retained, so multiple consumers read at their own offsets and can replay. RabbitMQ is a classic queue where a message goes to one worker and is acknowledged away. Q: A breaking change to an API version, managed well, should… - ship without warning and remove the old version immediately - go through a version bump with a documented migration path and a deprecation window for the old version correct: 1 explain: Lifecycle management means treating the API as a long-lived product. Breaking changes get a version bump, communication, and a sunset window so consumers can migrate. Q: An API will process credit-card payments. The most common way to stay manageable under PCI DSS is to… - handle raw card numbers yourself with strong encryption - route card data through a PCI-compliant payment provider so your API never touches the sensitive data and stays out of PCI scope correct: 1 explain: Most teams avoid PCI scope entirely by tokenizing through a compliant provider. If your API never sees the raw card data, the PCI burden falls on the provider, not you. Q: Under GDPR, a `DELETE /users/me` endpoint that removes the user record but leaves their data in five downstream services is… - compliant, because the main record was deleted - non-compliant — the right to erasure requires deleting downstream copies too correct: 1 explain: Erasure means the personal data is gone, everywhere it flows. A half-deletion that leaves copies in downstream services is a GDPR bug, not compliance. ```