Event-Driven Architecture: Building Async, Decoupled Systems

Modern backends rarely run as one synchronous request-response flow. Event-driven architecture lets services react to what already happened instead of waiting on each other, unlocking scalability, resilience, and decoupling. In this lesson you will master the core concepts and build a complete event-driven order pipeline with Apache Kafka.

1. Learning Objectives

By the end of this lesson, you will be able to:

• Explain the difference between events, commands, and plain messages
• Identify when event-driven architecture beats synchronous request-response
• Design topics, partitions, and event schemas for an event broker
• Implement producers and consumers with Apache Kafka
• Apply the outbox pattern to publish events reliably
• Compare event sourcing and CQRS with traditional CRUD designs
• Make consumers idempotent and handle failures with retries and dead-letter queues

2. Why This Matters

Imagine an e-commerce checkout. When a customer places an order, your system must update inventory, charge the payment provider, send a confirmation email, trigger shipping, and update analytics. Done synchronously, the customer waits while every downstream system responds, and one slow or failing service blocks the whole checkout. Netflix, Uber, and Amazon run event-driven backends precisely because of this problem: they publish facts like OrderPlaced once, and every interested service reacts independently, in its own time, at its own scale. Teams that master event-driven thinking build systems that absorb traffic spikes, degrade gracefully, and grow without rewiring every service.

3. Core Concepts

What Is an Event?

An event is a fact that already happened, expressed in the past tense: OrderPlaced, PaymentReceived, UserRegistered. Events are immutable, timestamped records with a unique id. Because they describe what happened rather than what should happen, they can be safely broadcast to any number of interested consumers.

Events vs Commands

A command is an intent or request directed at exactly one receiver: "please place this order". It may be rejected. An event is a statement of fact broadcast to everyone who cares: "the order was placed". Mixing the two is the most common design mistake. Use commands for APIs and request-response flows; use events for decoupled, fan-out integration between services.

The Event Broker

A broker such as Apache Kafka, RabbitMQ, or AWS SNS/SQS sits between producers and consumers. Producers publish events to a topic; consumers subscribe. The broker decouples the two: producers never know who is listening, and consumers can join or leave without affecting anyone else. Kafka goes further by ordering events within a partition and letting consumers replay history.

Event Sourcing

Instead of storing only the current state of an entity, event sourcing stores the full sequence of events that led to it. The current state is a projection you rebuild by replaying the events. This gives you a perfect audit trail and the ability to reconstruct state at any point in time. The trade-off: reads become more complex, which is why event sourcing is usually paired with CQRS.

CQRS

Command Query Responsibility Segregation splits the model in two: a write model that accepts commands and emits events, and a read model optimized for queries. The read model is often a denormalized projection, rebuilt from events, that makes reads fast and simple. You do not need event sourcing to use CQRS, but the two complement each other well.

The Outbox Pattern

The hardest problem in event-driven systems is the dual-write problem: writing to your database and publishing to the broker are two separate operations that can fail independently. The outbox pattern solves it by writing the event to an outbox table in the same database transaction as the business change. A separate relay process then publishes outbox rows to the broker and marks them sent. If the broker is down, the events wait safely in the database.

4. Hands-On Practice: An Event-Driven Order Pipeline

We will build a minimal but complete pipeline: an order service publishes OrderPlaced events to Kafka, and two independent consumers react — one updates inventory, one sends a notification. You need Docker and Python 3.10+.

Step 1: Start Kafka with Docker Compose

Create a docker-compose.yml with a single-node Kafka cluster:

version: "3.8"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.5.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
  kafka:
    image: confluentinc/cp-kafka:7.5.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Bring the cluster up and confirm both containers are healthy:

Step 2: Create the orders Topic

Kafka topics need to exist before producers write to them. Create orders with 3 partitions so multiple consumers can process in parallel while order is preserved per partition:

Step 3: Write the Producer

Install the Kafka client and write a producer that publishes a well-formed event. Note the key: all events for the same customer land in the same partition, preserving order per customer.

pip install confluent-kafka

import json
from confluent_kafka import Producer

conf = {"bootstrap.servers": "localhost:9092"}
producer = Producer(conf)

event = {
    "event_id": "evt-ord-1001",
    "type": "OrderPlaced",
    "occurred_at": "2026-07-31T10:00:00Z",
    "payload": {
        "order_id": "1001",
        "customer_id": "42",
        "total": 129.99
    }
}

producer.produce(
    topic="orders",
    key=str(event["payload"]["customer_id"]),
    value=json.dumps(event),
)
producer.flush()
print("Published", event["event_id"])

Step 4: Write the Consumers

Now two consumers subscribe to the same topic in the same consumer group. Kafka assigns each partition to one consumer in the group, so the workload is shared. Each consumer processes its events at its own pace.

import json
from confluent_kafka import Consumer

conf = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "inventory-service",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": True,
}
consumer = Consumer(conf)
consumer.subscribe(["orders"])

print("Inventory consumer waiting for events...")
while True:
    msg = consumer.poll(1.0)
    if msg is None:
        continue
    if msg.error():
        print("Error:", msg.error())
        continue
    event = json.loads(msg.value().decode())
    order_id = event["payload"]["order_id"]
    print(f"Inventory updated for order {order_id}")

Copy the file to consumer_notify.py, change the group.id to notification-service, and replace the last print with an email-sending log line. Run both consumers in two terminals, then publish several events:

Step 5: Guarantee Delivery with the Outbox Pattern

In production the producer should not write the business change and the event in two separate operations. Instead, write both in one database transaction, then let a relay publish:

BEGIN;
INSERT INTO orders (order_id, customer_id, total)
VALUES (1001, 42, 129.99);

INSERT INTO outbox (event_id, event_type, payload, created_at)
VALUES (
  'evt-ord-1001',
  'OrderPlaced',
  '{"order_id": 1001, "customer_id": 42}',
  NOW()
);
COMMIT;

-- A relay process then publishes outbox rows to Kafka
-- and deletes them once the broker acknowledges delivery.

If the broker is down, the events stay in the outbox table and the relay retries. This eliminates the dual-write problem without adding a distributed transaction.

5. Common Errors & Solutions

1. Topic not found when producing. The broker has auto.create.topics.enable=false, so writes fail with UNKNOWN_TOPIC_OR_PARTITION. Create the topic explicitly first (Step 2) or enable auto-creation for development.

2. Duplicate events on consumer restarts. Offsets are committed after processing by default, so a crash between processing and commit redelivers the event. Make consumers idempotent: store the event_id you have processed and skip repeats, or use an upsert keyed by business id.

3. Events processed out of order. Kafka only guarantees order within a partition. Use a stable key such as customer_id so related events land in the same partition, and avoid scaling a single logical stream across more consumers than it has partitions.

4. Consumer group rebalancing storm. Adding or removing consumers frequently triggers rebalances that pause the whole group. Start consumers slowly, use static group membership (group.instance.id) for long-lived workers, and keep poll intervals under max.poll.interval.ms.

5. Schema drift breaking consumers. Producers add a field and old consumers crash on unknown keys. Version your event schema and use a schema registry (Confluent Schema Registry or JSON Schema) so consumers validate against a shared, evolvable contract.

6. Summary Checklist

• I can explain events vs commands vs messages and when to use each
• I designed topics with a partition strategy that preserves per-entity ordering
• I wrote a Kafka producer that publishes schema-versioned events
• I wrote consumers that are idempotent and handle redelivery
• I used the outbox pattern to publish events reliably with my database writes
• I understand when event sourcing and CQRS add value, and when they add complexity
• I planned retries and a dead-letter strategy for poison messages

7. Practice Exercise

Build a UserRegistered pipeline: a producer publishes a user-registration event; three consumers react — one creates a welcome email job, one initializes a default profile, and one writes an audit log. Requirements:

• Use one topic and three consumer groups so every consumer sees every event
• Make each consumer idempotent using the event id
• Publish 100 events with 5 different user ids and verify each consumer processed all 100 exactly once
• Kill one consumer mid-stream and restart it; confirm no events are lost and none are double-processed

Bonus: add a dead-letter topic and route an intentionally malformed event to it instead of crashing the consumer.

8. Next Steps

You have now covered the full Backend Architecture arc: fundamentals, database scaling, caching, message queues, API design, microservices, and event-driven architecture. The natural next topic is handling failure in distributed systems — distributed transactions and the Saga pattern, which solve the data-consistency problem we kept hitting across these lessons. From there, the series moves into cloud computing, where you will deploy these exact patterns on AWS and Azure.