Redpanda — Kafka-Compatible Streaming Without ZooKeeper and JVM Overhead

The Operational Tax of Running Kafka

Apache Kafka is the default backbone for event streaming, and for good reason. But running it has always carried an operational tax: a JVM to tune, a separate coordination layer to babysit, and a fleet of brokers whose garbage-collection pauses show up as latency spikes in your p99.

Redpanda takes a different bet. It reimplements the Kafka protocol from scratch in C++, ships as a single binary, and drops the two dependencies that cause the most operational pain — ZooKeeper and the JVM. Your existing Kafka clients keep working, because Redpanda speaks the same wire API.

This guide walks the architecture that makes that possible, then gets hands-on: starting a cluster, driving it with the rpk CLI, producing and consuming, offloading to object storage with Tiered Storage, and planning a migration from Kafka. If you are still deciding whether streaming is even the right shape, our note on stream processing vs batch is a useful primer.

One Binary, Thread-per-Core, No JVM

The design choice that shapes everything else is the runtime. Redpanda is built on Seastar, an open-source thread-per-core C++ framework. Each application thread is pinned to a physical CPU core, and threads talk to each other by message passing instead of shared memory and locks.

That pinning avoids context switching and expensive locking. According to Redpanda’s architecture documentation, it also relies on Seastar’s custom memory allocator, and because Redpanda understands its own access patterns better than the OS does, it bypasses the Linux page cache and manages its own memory and disk I/O directly.

There is no JVM in the picture, so there is no garbage collector and no heap tuning. There is also no ZooKeeper or KRaft controller quorum to operate separately — a single self-contained binary is the broker, packaged with no external system dependencies.

# The Kafka mental model, but without the extra moving parts:
#
#   Apache Kafka                    Redpanda
#   ------------                    --------
#   JVM broker  + GC tuning         single C++ binary, no GC
#   ZooKeeper / KRaft quorum        built-in Raft, no separate service
#   page cache (OS-managed)         Redpanda-managed memory + disk I/O
#   thread pool + locks             thread-per-core (Seastar), message passing
#
# Same Kafka API on the wire  ->  existing producers/consumers just work.

The practical upshot is sizing. Redpanda’s docs state it can handle approximately one GB/sec of writes per core depending on workload, and since NVMe disks sustain over one GB/sec, it takes two cores to saturate a single NVMe disk. That density is why the same throughput can run on fewer, smaller nodes.

Raft All the Way Down

Kafka historically split responsibilities: ZooKeeper managed cluster metadata while an in-sync-replica protocol handled data replication. Redpanda uses the Raft consensus algorithm throughout the platform — to coordinate writing data to log files and to replicate that data across brokers.

Each partition is its own Raft group with a leader and followers. A write is acknowledged once a majority of the group has it durably, which gives you clear, well-understood consistency semantics for the distributed log without a bolt-on coordination service. Redpanda advertises Jepsen-verified results for data safety on its product page.

Because replication is Raft-based, the replication factor is expected to be an odd number — a majority quorum needs a tie-breaker. That constraint surfaces directly in the CLI, which we will hit in a moment.

Starting a Cluster with Docker Compose

The fastest way to a running cluster is the quickstart bundle, which brings up Redpanda plus Redpanda Console, the web UI for browsing topics and consumer groups. From the official quick-start guide:

mkdir redpanda-quickstart && cd redpanda-quickstart && \
  curl -sSL https://docs.redpanda.com/redpanda-quickstart.tar.gz | tar xzf - && \
  cd docker-compose && docker compose up -d

That single command downloads the Compose file and starts the containers detached. Everything from here is driven by rpk(“Redpanda Keeper”), the CLI that ships inside the binary. You can run it from inside a broker container:

# Cluster health and broker list
docker exec -it redpanda-0 rpk cluster info

# rpk reads brokers from a profile, or override per-command:
docker exec -it redpanda-0 rpk cluster info -X brokers=redpanda-0:9092

The -X flag sets configuration options inline — brokers, SASL user/pass, and sasl.mechanism among them. For repeated use, rpk profile create stores these so you do not repeat them on every call.

Topics, Produce, Consume

Topics are the unit of organization, exactly as in Kafka. The rpk topic create command takes -p/--partitions and -r/--replicas, and -c/--topic-config for per-topic properties. Remember the replication factor must be odd:

# Two topics, 20 partitions each, replication factor 3, log compaction on
docker exec -it redpanda-0 rpk topic create \
  -p 20 -r 3 \
  -c cleanup.policy=compact \
  orders payments

# TOPIC     STATUS
# orders    OK
# payments  OK

Producing and consuming are symmetric subcommands. rpk topic produce reads records from stdin; rpk topic consume reads them back, with --num to limit the count and --offset to choose a starting point.

# Produce: type a line, press Enter, Ctrl+C to stop
echo '{"id": 1, "amount": 4200}' | \
  docker exec -i redpanda-0 rpk topic produce orders

# Consume one record from the start of the topic
docker exec -it redpanda-0 rpk topic consume orders --num 1 --offset start

# Consume as a named group so offsets are tracked
docker exec -it redpanda-0 rpk topic consume orders --group billing-svc

Inspect what you built with rpk topic describe orders for partition leaders and replica assignments, or rpk topic list for an overview. Because this is the Kafka API, your application code does not use rpk at all — it uses whatever Kafka client library you already have.

Your Existing Kafka Clients, Unchanged

The whole value proposition rests on protocol compatibility. Point a standard Kafka client at Redpanda’s bootstrap address and it behaves like any other Kafka broker. Here is the confluent-kafka Python client — the same code you would write against Apache Kafka:

from confluent_kafka import Producer, Consumer

producer = Producer({"bootstrap.servers": "localhost:19092"})
producer.produce("orders", key="1", value='{"amount": 4200}')
producer.flush()

consumer = Consumer({
    "bootstrap.servers": "localhost:19092",
    "group.id": "billing-svc",
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["orders"])
msg = consumer.poll(timeout=5.0)
if msg and not msg.error():
    print(msg.value())
consumer.close()

Nothing in that snippet mentions Redpanda. The same is true for Java, Go, and Node clients, for Kafka Streams applications, and for connectors — which is exactly what makes an incremental migration realistic. If your topics carry structured events governed by a registry, our write-up on event-driven architecture with Kafka and Schema Registry transfers directly.

Tiered Storage: Cheap Retention on Object Stores

Retaining months of history on local NVMe is expensive. Redpanda’s Tiered Storage offloads older log segments to object storage — S3, GCS, or Azure Blob — while keeping the recent tail on disk. Consumers reading historical offsets fetch transparently from the object store.

It is enabled per topic with two properties: redpanda.remote.write to upload segments, and redpanda.remote.read to serve reads from remote. You set them at creation time or alter them later:

# Enable Tiered Storage on a new topic
docker exec -it redpanda-0 rpk topic create events \
  -p 12 -r 3 \
  -c redpanda.remote.write=true \
  -c redpanda.remote.read=true

# Or turn it on for an existing topic
docker exec -it redpanda-0 rpk topic alter-config events \
  --set redpanda.remote.write=true \
  --set redpanda.remote.read=true

The cluster-level object-store credentials (bucket, region, and access keys such as ${AWS_ACCESS_KEY_ID}) are configured once via cluster properties. Once retention exceeds local disk, older data lives in the bucket, and you pay object-store prices for the long tail instead of block-storage prices. Redpanda’s product page frames this as part of its cost story, claiming roughly 6x cost reduction compared to Apache Kafka — a vendor figure worth validating against your own workload.

Migrating from Apache Kafka

Because the wire protocol matches, migration is a data-movement problem, not a rewrite. The lowest-risk path runs both systems side by side and mirrors traffic. Redpanda supports rpk for administration, and MirrorMaker 2 works because both ends speak Kafka.

A pragmatic sequence: stand up Redpanda, replicate topics with MirrorMaker 2, cut consumers over group by group while producers still write to Kafka, then flip producers once consumers are healthy on Redpanda. This is the same dual-write discipline we describe for database migrations without downtime — expand, migrate, contract.

# MirrorMaker 2 replicating Kafka -> Redpanda (mm2.properties excerpt)
clusters = source, target
source.bootstrap.servers = kafka-broker:9092
target.bootstrap.servers = redpanda-0:9092

source->target.enabled = true
source->target.topics = orders,payments,events

# Run with the standard Kafka tooling:
#   connect-mirror-maker.sh mm2.properties

Validate consumer-group offsets after each cutover, and keep the source cluster until you have run through at least one full retention window on Redpanda. For CDC pipelines feeding these topics, the same Debezium connectors continue to work — see our change data capture guide for the producer side.

When Redpanda Earns Its Place

Redpanda is most compelling when operational simplicity and latency matter more than Kafka’s vast plugin ecosystem. Teams tired of JVM tuning, ZooKeeper coordination, and large broker fleets get a single binary that scales up on dense hardware before it has to scale out.

It is less obviously worth switching if you are deeply invested in JVM-based tooling that assumes internals beyond the Kafka API, or if your Kafka cluster is already well-run and boring. “Boring and working” is a feature; migrate for a concrete pain, not novelty.

Production checklist

• Prove it on your workload. Benchmark with your real message sizes, partition counts, and retention before trusting any vendor 10x or 6x figure.
• Keep replication odd. Raft needs a majority — use 3 replicas in production so a single broker loss is survivable.
• Size by cores and NVMe. Budget roughly one core per GB/sec of writes and remember two cores saturate an NVMe disk.
• Enable Tiered Storage early. Turn on remote read/write for long-retention topics so history lands in object storage, not block storage.
• Reuse your clients and connectors. The Kafka API means no application rewrite; validate Schema Registry and Connect against Redpanda in staging.
• Migrate incrementally. Mirror with MirrorMaker 2, cut consumers before producers, and keep the source cluster for one full retention window.

Redpanda’s bet is that the streaming layer should be one process that behaves like Kafka on the wire and like a purpose-built C++ system underneath — no JVM to tune, no ZooKeeper to operate. If your streaming platform’s biggest problems are operational rather than functional, that trade is worth a proof of concept.