20 — Data and Messaging Patterns: CQRS, Event Sourcing, Pub/Sub, and Competing Consumers

"Just use a queue" was how I conflated data and messaging patterns, which live at different layers. Writing them down separated the concerns: data patterns reshape how data is stored and read (CQRS, event sourcing, materialized views, index tables), and messaging patterns shape how work moves between components (pub/sub, competing consumers, queue-based load leveling, claim check). [1][2] Both families solve scale problems, but at different layers.

The framing that landed is to ask which layer has the bottleneck. If the bottleneck is "my read shape does not match my write shape," the data-pattern family applies. If the bottleneck is "producers and consumers need to be decoupled," the messaging-pattern family applies. Most large systems use both — a messaging pattern to move events around, and a data pattern to reshape them for efficient reads.

Data patterns: reshape for reads and writes

These patterns accept that one model rarely serves both writes and reads well at scale, and they split or pre-compute accordingly [1]:

• CQRS (Command Query Responsibility Segregation) separates read and update operations for a data store. It maximizes performance, scalability, and security by letting the read and write sides evolve independently, and prevents update commands from causing merge conflicts at the domain level [3]. The write side handles commands; one or more read models are built from the events the write side emits.
• Event sourcing stores the full series of actions taken on data in an append-only log, rather than just the current state. The log is the system of record and can be used to materialize domain objects. It simplifies complex domains, improves performance and scalability, and maintains a full audit trail that enables compensating actions [4]. Combined with CQRS, the event log is the write side and the read models are materialized projections of it.
• Materialized view generates prepopulated views over data when the data is not ideally formatted for the required queries, supporting efficient querying and improving performance [5]. This is the read-model half of CQRS — a denormalized table built for a specific query shape.
• Index table creates indexes over fields frequently referenced by queries, improving query performance by letting applications locate data more quickly [6]. The textbook fix for "this query is slow because it scans the whole table."
• Sharding (covered in the database notes) horizontally partitions data across servers for performance, scalability, and availability [7].

The way of thinking for the data family: stop trying to serve every read from the same model that accepts writes. Write once, project many times into read-optimized shapes. The cost is eventual consistency between the write and the read models (a projection lags the log), and the operational complexity of maintaining the projections.

Messaging patterns: move work between components

These patterns decouple producers from consumers using a messaging substrate [2]:

• Publisher-Subscriber (pub/sub) lets an application announce events to multiple interested consumers asynchronously, without coupling senders to receivers [8]. The publisher does not know who is listening; each subscriber gets its own copy. Good for fan-out — one event triggers many independent reactions.
• Competing consumers enables multiple concurrent consumers to process messages received on the same channel, optimizing throughput, improving scalability and availability, and balancing the workload [9]. This is the pattern behind a worker pool draining one queue — parallelism for throughput.
• Queue-based load leveling uses a queue as a buffer between a task and a service to smooth intermittent heavy loads that could cause the service to fail or time out [10]. Same idea as the asynchronism notes — the queue absorbs rate mismatch.
• Priority queue prioritizes requests so higher-priority ones are processed before lower-priority ones, for applications offering different service-level guarantees [11].
• Claim check splits a large message into a claim check and a payload — send the (small) claim check to the message bus and store the (large) payload in external storage, protecting the bus and clients from being overwhelmed [12]. Essential when messages carry large binaries.
• Choreography has each component participate in the decision-making about a workflow, instead of relying on a central orchestrator [13]. The counterpoint is orchestration (a central scheduler); choreography is more decoupled but harder to follow.
• Sequential convoy executes a series of dependent tasks in a specific order, handling errors or failures during execution [14]. Used when order matters — workflows and transactions.
• Asynchronous request-reply decouples backend processing from a frontend host, where the backend is asynchronous but the frontend still needs a clear response [15].

How I use this

The decision is about which layer is the bottleneck. When read shapes diverge from write shapes — heavy read load with many query patterns against one write model — I reach for CQRS with materialized views, and event sourcing if an audit trail or temporal queries are also required. When producers and consumers need decoupling — fan-out, load leveling, parallel workers — I reach for the messaging family: pub/sub for one-to-many, competing consumers for parallelism, claim check for large payloads. The two families compose naturally: an event-sourced write side _publishes events_ via pub/sub, and competing consumers materialize them into read models. The discipline is to introduce these patterns only when the simpler single-model, synchronous design has stopped working — they add real complexity, and adopting them too early is a common over-engineering mistake.

References

[1] Microsoft, "Data management patterns," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/category/data-management

[2] Microsoft, "Messaging cloud patterns," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/category/messaging

[3] Microsoft, "CQRS pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/cqrs

[4] Microsoft, "Event Sourcing pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/event-sourcing

[5] Microsoft, "Materialized View pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/materialized-view

[6] Microsoft, "Index Table pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/index-table

[7] Microsoft, "Sharding pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/sharding

[8] Microsoft, "Publisher-Subscriber pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber

[9] Microsoft, "Competing Consumers pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/competing-consumers

[10] Microsoft, "Queue-Based Load Leveling pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling

[11] Microsoft, "Priority Queue pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/priority-queue

[12] Microsoft, "Claim Check pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/claim-check

[13] Microsoft, "Choreography pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/choreography

[14] Microsoft, "Sequential Convoy pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/sequential-convoy

[15] Microsoft, "Asynchronous Request-Reply pattern," Azure Architecture Center. [Online]. Available: https://learn.microsoft.com/en-us/azure/architecture/patterns/async-request-reply