I reread Jay Kreps' essay The Log: What every software engineer should know about real-time data's unifying abstraction while we were rebuilding our event pipeline. The paper's thesis is simple yet profound: model your system as an immutable, append‑only log and complexity starts to fall away.
Image credit: Apache Software Foundation
The log as the source of truth
Kafka's commit log is a data structure that decouples producers from consumers. Each topic is partitioned and replicated, giving us parallelism and fault tolerance. Producers append bytes; consumers track offsets independently. The log never changes, only grows.
This idea eliminated whole classes of bugs in our system. Services stopped sharing mutable state and instead wrote facts—"user signed in", "device offline"—to topics. Downstream services materialized views in their own datastores, replaying from offset 0 whenever new indexes or backfills were required.
Retention and compaction
The paper's discussion on retention policies inspired our usage of tiered storage. Hot data lives on NVMe, while older segments migrate to S3 via Kafka's object storage integration. For entities that evolve, such as user profiles, log compaction keeps only the latest record per key, trimming history without losing meaning.
Exactly-once semantics
Idempotency is tricky when consumers crash mid‑process. Kafka's transactional APIs, combined with the outbox pattern, gave us exactly-once guarantees. Producers write events and offsets in a single transaction; consumers commit offsets to a transactionally consistent store. If a consumer dies, its work is retried without duplication.
producer.initTransactions();
producer.beginTransaction();
producer.send(record);
producer.sendOffsetsToTransaction(offsets, groupId);
producer.commitTransaction();Operational insights
Running Kafka taught us to treat partitions as the unit of scale. Skewed keys cause hot partitions, so we introduced hashing and monitored per‑partition bytes. kafka-reassign-partitions.sh became part of our runbooks, and we automated broker healing with Cruise Control.
Beyond Kafka
The log abstraction appears in other papers too: Amazon's Dynamo and Facebook's LogDevice both rely on append-only structures. Understanding the Kafka paper made those designs easier to grok and convinced us that event logs are the backbone of modern distributed systems.
Reading the paper didn't just clarify Kafka; it shifted how we design services. Whenever a new feature is proposed we ask: what's the log of facts, and who consumes it? Starting from that question has saved us from premature complexity more than once.