Kafka architecture—a deep dive

Kafka architecture

Apache Kafka^® is an open-source distributed data streaming platform that many companies use for high-performance data pipelines and streaming analytics. It's often described as a "distributed commit log," a system that provides a durable, ordered record of events you can replay to rebuild application state consistently. Kafka supports mission-critical use cases with guaranteed per-partition ordering, durable replication, and exactly-once processing semantics.

This article walks through Kafka's architecture as it exists today: its core building blocks, how metadata is managed, and the design principles that make the system scale.

Summary of key Kafka architecture concepts

Apache Kafka’s evolution

Kafka was first invented at LinkedIn in 2010. By 2011, it had entered incubation at the Apache Software Foundation and graduated to a Top-Level Project in 2012. It has consistently evolved over time since then. This document is a point-in-time survey of Apache Kafka, updated as of version 4,3. Users should check to see the latest available version.

Over time, new features and capabilities are added to Apache Kafka through a Kafka Improvement Proposal (KIP). A list of adopted and currently-under discussion KIPs can be found at the Apache Kafka Confluence page. Where appropriate, specific KIPs have been cited in this article.

Data streaming vs. event streaming

In earlier Kafka-oriented articles, the terms “events” and “event streaming” were used. However, this often confuses readers, conflating data streaming technology such as Apache Kafka with live video or audio streaming, say, of conference talks, music concerts, or sporting events. 

In this article, as across much of the industry, the term “data streaming” has supplanted “event streaming” and will be used to disambiguate Kafka-like services from video- and audio-oriented live or on-demand streaming. “Data streams” will be used instead of “events.”

However, there are still significant bodies of work that rely on the term “events,” or related terms such as event-based or event-driven architectures (EDA).

Kafka architecture components

Broker

A single Kafka server is called a broker. A broker receives data streams from producers, assigns them offsets, appends them to disk, and services consumers' fetch requests. Depending on hardware and configuration, a single broker can comfortably handle thousands of partitions and millions of messages per second.

Brokers operate as part of a cluster. A subset of brokers (or dedicated nodes) forms the controller quorum, and one of them (the active controller) handles administrative operations such as assigning partitions and responding to broker failures.

Message (event)

A message is a key-value pair, serialized to bytes before it's written to disk.

Keys generally route messages to a partition. They're usually strings or integers identifying an entity (a user, order, or device) and don't need to be unique. A null key causes Kafka to distribute the message across partitions rather than route it deterministically.

Values carry the event payload, anything from a short string to a deeply nested object, or nothing at all. The default MAX payload size for a Kafka message is 1 MB (with compression). Optimal performance can usually be achieved with smaller payload sizes (between 100 bytes to 10 kilobytes). Some use cases may even set larger payload sizes (up to 50 MB or more). While the latter degrades performance, it may be required to support certain use cases. Users can sometimes also break large payloads into multiple smaller chunks.

Every message also carries a timestamp and, optionally, a set of headers. These are string key-value pairs used for metadata such as schema ID markers, provenance, or routing hints.

Topic

Messages are categorized into topics. These are named, ordered, append-only event logs. Multiple producers can write to the same topic, and multiple independent consumers or consumer groups can read from it without affecting one another. Reading a message doesn't remove it: unlike a traditional message queue, Kafka retains events for a configurable period (seven days by default) regardless of how many times they've been read.

Partition

Each topic is divided into one or more partitions, which are the actual unit of the "commit log." A partition is a single append-only log, written in order and read in order. Partitions are numbered starting at 0, and the number of partitions is set per topic (and can be increased).

Splitting a topic into partitions is what makes Kafka horizontally scalable: each partition can live on a different broker, and each can be replicated independently, so a topic's throughput and durability aren't capped by any single machine.

Partition offset

Every message in a partition gets a monotonically increasing integer offset, starting at 0. Offsets are never reused, even after old messages are deleted by retention. A keyless message lands in a partition via a round-robin or sticky-partition strategy; a keyed message is hashed to consistently land in the same partition every time, which is how Kafka guarantees ordering within a partition, but not across partitions in a topic.

Partition replication: leaders, followers, and ISRs

Each partition is owned by one broker, the partition leader, which handles all reads and writes for that partition. If the topic's replication factor is greater than 1, copies of the partition (aka. followers) are held on other brokers.

Followers that are sufficiently caught up with the leader form the partition's in-sync replica (ISR) set. Only replicas in the ISR are eligible to be promoted to leader if the current leader fails.

A more recent refinement, Eligible Leader Replicas (ELR) (KIP-966), further tightens this: it tracks which out-of-sync replicas are still safe to promote in a last-resort failover, reducing the window during which a cluster could silently lose committed data during a correlated outage. This matters more as organizations push min.insync.replicas and acks settings toward stronger durability guarantees.

Tiered storage

Kafka 3.9 made tiered storage (KIP-405) generally available: brokers can offload older log segments to object storage (Amazon S3, GCS, Azure Blob, or other S3-compatible stores) while keeping recent segments on local disk. Kafka 4.3 continues to build out this path. For example, new configuration for follower fetches directly from the tiered offset, and dedicated admin-client and durability settings for the internal __remote_log_metadata topic that tracks what's been offloaded.

Unlike messaging queues, reading a message from a topic doesn't delete it. Messages can be read as often as needed by multiple different applications. The data written to topics is durable and retained on the disk for seven days by default. However, this behavior can be configured.

The effect is that retention and local disk size become independent variables: a topic can retain data for months or years without every broker needing months or years of local disk, which changes how teams think about capacity planning and cost.

Reading from and writing to Kafka

Producers

Producers are client applications that publish data streams. They take a key-value pair, serialize it, decide which partition it should land in, and send it (batched and compressed) to the partition leader. Traditionally in Kafka, all writes for a partition go through its leader; producers would not write directly to consumers. However, this is changing with new “brokerless” or “diskless” architectures (see below).

Consumers

Consumers subscribe to topics and read events in offset order, per partition, by polling the broker at intervals. Order is guaranteed within a partition, not across the partitions of a topic a consumer is reading from concurrently. Consumers track their own read position (the offset) so they can stop and resume without losing their place. Reading doesn't delete a message. Any number of independent consumers can read the same data.

By default, all reads go through the partition leader, though Kafka also supports follower fetching (KIP-392), allowing consumers to read from a nearby replica. This is useful for reducing cross-zone or cross-region network costs in multi-AZ deployments.

Diskless and brokerless architectures

Everything described above is still the classic Kafka write path: a partition leader owns local disk, followers replicate that data over the network, and producers and consumers talk to whichever broker currently holds leadership for a given partition. That's still what ships in stock Kafka, but it's no longer the only architecture in the Kafka ecosystem, and cloud infrastructure bills are the reason why. 

Users consistently report that cross-AZ replication traffic (copying every write to followers in other availability zones) accounts for the vast majority of a cloud Kafka deployment's infrastructure costs, often 80% or more, since cloud providers bill separately for inter-zone data transfer.

Two related but distinct terms have emerged to describe architectures that avoid that cost:

• Brokerless architectures eliminate the traditional stateful broker entirely. WarpStream pioneered this model in 2023 (WarpStream was later acquired by Confluent): a fleet of stateless "agents" speaks the Kafka protocol but holds no local disk and does no inter-broker replication. Every write goes straight to object storage, and durability comes from the object store's own cross-AZ replication rather than from Kafka's leader/ISR model.
  
  
• Diskless topics are Kafka's own upstream answer to the same problem, defined in KIP-1150. The Kafka community accepted KIP-1150 on March 2, 2026, but it's what the community calls a "motivational" or umbrella KIP. It establishes that the project wants this capability and sketches its shape, without shipping any code. The follow-on implementation KIPs — KIP-1163 (Diskless Core), KIP-1164 (Diskless Coordinator), and KIP-1165 (Object Compaction) — were still under active discussion and not yet accepted as of this writing, so diskless topics are not available in any stock Apache Kafka release today, including 4.3.

As designed, diskless topics would be leaderless (any broker can accept a produce or fetch request for any partition), would batch writes from many partitions into shared log segment objects in object storage, and would use a pluggable coordinator to assign offsets and track batch locations. Opt-in per topic, so latency-sensitive topics could stay on the classic disk-and-replication path in the same cluster. The trade-off, based on early published projections, is tail latency: hundreds of milliseconds for an object storage round trip on the write path, versus single-digit-millisecond scale for a classic local disk write.

It's worth distinguishing these diskless or brokerless options from tiered storage (covered above): tiered storage only offloads already-sealed, cold segments to object storage while the active write path stays on replicated local disk. Diskless removes the local disk from the active write path. This changed how new data is written and replicated, not just to where old data eventually rests.

Since the upstream implementation is still years away from general availability, according to most public projections, teams that want this model today reach for systems that already ship it: WarpStream, Aiven's open-source Inkless fork, AutoMQ, Bufstream, and StreamNative Ursa are the most commonly cited.

Good to know: Redpanda takes a middle path with Cloud Topics (generally available in Redpanda Streaming 26.1, March 2026): message payloads stream directly to object storage on the write path, but metadata, offsets, ACLs, and Raft consensus stay on local NVMe rather than being handed off to an external coordinator. 

This aims to capture a large share of the networking cost savings of a fully diskless design without moving the "brains" of the system off the same battle-tested consensus engine that governs standard topics, and without introducing an external metadata service as a new single point of failure. Cloud Topics coexist with standard Redpanda topics and Tiered Storage, configurable per topic.

Consumer groups and the modern rebalance protocol

A single consumer eventually gets overwhelmed as partition count and throughput grow. Consumer groups solve this: a set of consumers coordinates to divide a topic's partitions, with each partition assigned to exactly one group member at a time. If a member fails, its partitions are reassigned to the survivors.

Historically, this coordination used the "classic" rebalance protocol, which paused all consumption across the whole group ("stop-the-world" rebalancing) whenever membership changed. This changed in Kafka 2.4 with Incremental Cooperative Rebalancing (KIP-429). Kafka 4.0 introduced the Next Generation Consumer Rebalance Protocol (KIP-848), yet “classic” rebalancing still remained the default. It moves partition assignment logic to the group coordinator (a broker-side component) and reassigns only the partitions that actually need to move, so a single consumer joining or leaving no longer stalls the entire group. 

As of Kafka 5.0, the Next Generation Consumer Rebalance Protocol will become the default, and with a new KIP (KIP-1274), the classic protocol is formally set for deprecation in a future major release. New deployments should build on the new protocol rather than the classic one.

The number of partitions in a topic still caps the number of consumers a group can usefully support: a five-partition topic can support at most five consumers in one group, with any extra sitting idle.

Share groups: Kafka as a queue

Consumer groups assign whole partitions to consumers. That model is great for ordered, high-throughput streaming, but awkward for workloads that look more like a task queue, where you want many workers pulling individual records with per-record acknowledgment and retries, regardless of partition boundaries.

Share groups (KIP-932), introduced as early access in Kafka 4.0 and matured through 4.1–4.3, add that model natively. Multiple consumers in a share group can read from the same partition concurrently; records are delivered and acknowledged individually rather than by committing an offset watermark, and unacknowledged records are automatically redelivered. Kafka 4.3 adds further group-level configuration for share groups (KIP-1240) and coordinator buffer-size controls shared with regular consumer groups (KIP-1196), reflecting that this is now a maturing, first-class consumption model rather than a niche add-on, which was useful for teams who previously reached for a separate queue product (SQS, RabbitMQ) alongside Kafka just to get per-message acknowledgment semantics.

A minimal producer and consumer, in current form

While there are hundreds of existing connectors for Kafka producers and consumers, users may sometimes need to write their own. For example, a unique internal microservice, a bespoke application, or a rare data source or destination. Let’s look at what would be required to create a simple producer or consumer application.

// Producer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    producer.send(new ProducerRecord<>("test-topic", "Simple string message from the producer!"));
}

// Consumer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("test-topic"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("key = %s, value = %s%n", record.key(), record.value());
        }
    }
}

Producers and consumers are fully decoupled: neither knows the other exists, and neither's speed or failure affects the other directly.

From ZooKeeper to KRaft: how Kafka manages metadata now

For most of its history, Kafka depended on Apache ZooKeeper™, a separate distributed coordination service, to track cluster membership, elect partition leaders, and store topic configuration. Operating Kafka meant operating two distributed systems: Kafka itself and the ZooKeeper ensemble it depended on.

KIP-500 proposed removing that dependency by having a subset of Kafka brokers run the metadata quorum themselves, using the Raft consensus algorithm. This mode is called KRaft (Kafka Raft). KRaft reached general availability in Kafka 3.3; ZooKeeper-based deployments were deprecated in Kafka 3.5–3.9; and as of Kafka 4.0 (released in 2025), ZooKeeper has been removed from the codebase entirely. Every Kafka 4.x cluster (including 4.3) runs on KRaft. There is no ZooKeeper mode left to fall back to; clusters still on ZooKeeper must migrate before upgrading past 3.9.

Under KRaft:

• A small set of brokers are designated as controllers, forming a Raft quorum (typically three or five nodes for fault tolerance).
• Cluster metadata (topic configs, partition assignments, ACLs, broker membership) is written to an internal, replicated metadata log rather than to a separate ZooKeeper znode tree.
• One controller is elected active controller (the Raft leader for metadata) and propagates changes to the rest of the cluster; the others stand by as hot replicas of the metadata log.
• Controllers can run in combined mode (a node acts as both a broker and a controller, common for smaller or dev clusters) or in isolated mode (dedicated controller-only nodes, recommended for larger production clusters, as they isolate metadata operations from data-plane load).

The practical effect: one fewer distributed system to deploy, patch, and monitor; faster controller failover and higher partition-count ceilings, since the metadata log scales more efficiently than ZooKeeper's znode model did.

Kafka 4.3 continues to refine this metadata path rather than replace it. For example, new configurations for controller quorum fetch and snapshot sizing give operators finer control over how much metadata traffic the Raft quorum handles at once, which matters more as cluster and topic counts grow.

Schema Registry

A schema registry provides a centralized repository for managing and validating the schemas used by topic message data, and for serializing and deserializing that data over the network. One thing worth clearing up: Kafka does not ship a schema registry. It’s a separate, pluggable component, and you have several implementations to choose from, including free and open-source options:

Apicurio Registry: open source (Apache 2.0), backed by Red Hat.

Karapace: open source (Apache 2.0), API-compatible with the Confluent registry.

Confluent Schema Registry: free to use under the Confluent Community License (source-available).

AWS Glue Schema Registry: a managed option for AWS-centric stacks.

Redpanda takes a simpler route: a Schema Registry is built into every cluster and speaks the same API your Kafka clients already expect, so there’s no extra service to deploy or operate.

Options aside, the Schema Registry ensures that a message's schema is valid and compatible with the expected schema for that topic. It provides the following services:

• Lets producers and consumers agree on a well-defined data contract
• Enforces explicit compatibility rules as schemas evolve
• Reduces payload size on the wire by sending a schema ID instead of the full schema definition
• Serves as a basis for translating Kafka topic records into Apache Iceberg™ table records.

Security in Kafka architecture

Kafka supports SASL (Simple Authentication and Security Layer) for authenticating clients (producers, consumers, and other components) before they can interact with the cluster, alongside TLS for encryption in transit. For OAuth-based authentication, Kafka 4.3 adds support for OAuth client assertions with the client_credentials grant type (KIP-1258), improving compatibility with modern identity providers that expect assertion-based client auth rather than a shared client secret.

Access Control Lists (ACLs) provide fine-grained authorization on top of authentication, controlling which principals can produce to or consume from specific topics, create or delete topics, or view cluster metadata like the list of topics or consumer groups.

Kafka in the real world

Kafka serves as the circulatory system of much modern data infrastructure, carrying events between applications, services, and analytical systems through a single, consistent interface.

Use cases

User activity tracking. Site activity (page views, searches, or other user actions) can be published to central topics, with one topic per activity type. Those activity streams can then land in cloud data warehouses like Snowflake, BigQuery, or Redshift, or in data lakes and lakehouses built on open table formats such as Iceberg or Delta Lake, for reporting and offline analysis.

Log and event aggregation. Kafka provides a durable, replicated abstraction over log and event data, with lower end-to-end latency than older log-shipping systems.

Metrics collection. Distributed applications publish operational metrics to Kafka, which centralizes them for monitoring and alerting pipelines.

Change data capture and event sourcing. Database changes are published to Kafka as an ordered log, which downstream systems replay to build derived views (for example, an analytical database wanting to shadow updates occurring in a transactional system of record). 

Stream processing. Frameworks, including Apache Flink^®, Apache Spark™, and Kafka's own Kafka Streams library, consume Kafka topics to filter, aggregate, join, and transform events in real time, then write the results back to Kafka or to other systems.

Streaming into the lakehouse. Increasingly, Kafka topics are written directly into open table formats like Apache Iceberg, so the same data stream that feeds real-time applications also lands as queryable historical data, narrowing the gap between "the data ecosystem's circulatory system" and the lakehouse it feeds.

Final thoughts

Kafka's architecture is distributed and fault-tolerant, and since Kafka 4.0, it’s no longer split across two separate coordination systems. Removing ZooKeeper simplified operations meaningfully, but Kafka is still a JVM-based system with its accompanying heap memory tuning requirements and performance constraints. Kafka is now a system with real operational depth: controller quorum sizing, tiered storage, rebalance protocol choices, and share-group tuning are all decisions operators now own directly, without an external coordination layer to blame or lean on.

Redpanda Streaming is a Kafka API–compatible streaming data platform built as a single C++ binary with no JVM and no separate coordination service. Redpanda's Raft-based consensus has been built into the core since before KRaft existed. Every broker is identical in how it handles message participation, coordination, schema management, and HTTP ingress/egress. The goal is the same: durability and ordering guarantees that applications already expect from the Kafka API, with less infrastructure running underneath it. 

You can self-host it on-premises, consume it as a fully managed serverless offering, or run it fully managed in your own cloud account as a Bring Your Own Cloud (BYOC) deployment. You can get a cluster running in seconds rather than days. Take it for a free spin!

[CTA_MODULE]