Kafka Interview Questions for Data Engineers (2026)

Kafka is a distributed, append-only commit log for publishing and subscribing to streams of events. Unlike a traditional broker, messages are not deleted on consumption — they are retained for a configured period, so multiple independent consumer groups can replay the same data. Combined with partitioned horizontal scaling and sequential disk I/O, this makes it the standard backbone for CDC, event-driven microservices, and streaming ETL.

A topic is a named stream of events, split into partitions that are the unit of both parallelism and ordering. Each partition is an ordered, immutable log where every record gets a monotonically increasing offset. Consumers track their position by committing offsets, which is how Kafka supports replay — you can seek back to any retained offset and reprocess.

A consumer group is a set of consumers sharing a group.id that divide a topic's partitions among themselves, with each partition assigned to exactly one consumer in the group. This gives queue semantics within a group and pub-sub semantics across groups. If consumers outnumber partitions, the extras sit idle — which is why partition count caps your consumption parallelism.

Retention is time- or size-based per topic (retention.ms, default 7 days, or retention.bytes) and completely independent of consumption — reading never deletes anything. A topic can alternatively use cleanup.policy=compact to keep the latest record per key indefinitely. This decoupling is what lets you re-run a broken pipeline simply by resetting consumer offsets.

acks=0 is fire-and-forget with no broker acknowledgement — fastest, but can silently lose data. acks=1 waits only for the partition leader to write locally, so data is lost if the leader dies before followers replicate. acks=all waits until all in-sync replicas have the record; combined with min.insync.replicas=2 it is the standard durability setting, and it has been the default since Kafka 3.0.

The in-sync replica set is the leader plus the followers that are fully caught up within replica.lag.time.max.ms. With acks=all, a write is acknowledged only after every ISR member has it; min.insync.replicas sets the floor — with replication factor 3 and min.insync.replicas=2, you tolerate one broker failure, and the producer gets NotEnoughReplicasException rather than a silent under-replicated write if a second one drops out. Consumers only see records up to the high watermark, i.e., what the full ISR has replicated.

At-least-once is the default: producer retries after an ack timeout can write duplicates, and a consumer that commits offsets after processing will reprocess records if it crashes mid-batch. At-most-once comes from the opposite choices — a producer with retries disabled, or a consumer that commits before processing, so a crash loses the in-flight records. In interviews, tie each mode to the exact failure window rather than just naming the three semantics.

With enable.idempotence=true, the broker assigns the producer a PID, and the producer attaches a per-partition sequence number to every batch. On a retry, the broker recognizes a sequence it has already persisted and discards the duplicate; a gap raises OutOfOrderSequenceException. It is on by default since Kafka 3.0, but only covers a single producer session writing to a single partition — surviving restarts or spanning topics requires transactions.

Ordering is guaranteed only within a partition, never across a topic. You get per-entity ordering by keying messages (say, user_id), because the default partitioner hashes the key with murmur2 to a fixed partition. With idempotence enabled you can keep max.in.flight.requests.per.connection at up to 5 and the broker still preserves order across retries; without it, in-flight greater than 1 plus retries can reorder batches.

With cleanup.policy=compact, background cleaner threads rewrite log segments keeping at least the latest value for each key, so the topic becomes a snapshot of current state rather than a bounded history. A record with a null value is a tombstone that deletes the key, retained for delete.retention.ms (24 hours by default) so consumers observe the delete. Use it for changelog-style data — CDC snapshots, Kafka Streams state store changelogs, and Connect's internal config and offset topics.

Rebalances fire when members join or leave, when a consumer misses session.timeout.ms heartbeats or exceeds max.poll.interval.ms between polls, or when partition count changes. The classic eager protocol revokes every partition from every member — a stop-the-world pause — while CooperativeStickyAssignor moves only the partitions that actually need to move, letting the rest keep consuming. Static membership via group.instance.id additionally avoids rebalancing on quick restarts, and Kafka 4.0's KIP-848 protocol makes this coordination broker-driven and incremental by default.

Connect is Kafka's framework for moving data between Kafka and external systems using pre-built source and sink connectors — Debezium for CDC from MySQL or Postgres, sinks for S3, Elasticsearch, or a warehouse. In distributed mode a cluster of workers splits connectors into parallel tasks, and offsets, configs, and status live in internal Kafka topics, so pipelines survive worker failure without custom code. Write a consumer only when you need business logic; for pure ingestion or egress, Connect with SMTs for light transforms is the right answer.

Exactly-once combines idempotent producers with transactions: a producer configured with a transactional.id wraps its output records and the consumed offsets in one atomic unit via sendOffsetsToTransaction(), so the consume-process-produce cycle commits or aborts as a whole. Downstream consumers must set isolation.level=read_committed to skip aborted records; in Kafka Streams this is just processing.guarantee=exactly_once_v2. The guarantee holds within Kafka-to-Kafka pipelines — once you write to an external system, you need idempotent or transactional sinks to extend it.

If all ISR replicas die, the partition is unavailable until either an ISR member returns or, with unclean.leader.election.enable=true, an out-of-sync follower is elected leader — restoring availability but permanently losing every message the old leader had that the follower did not. It is false by default because Kafka favours durability over availability here. The complete durability posture is RF=3, min.insync.replicas=2, acks=all, unclean election off — interviewers want the tradeoff articulated, not just the flag named.

Producers serialize against a schema registered in the registry and prepend a magic byte plus a 4-byte schema ID to each message; consumers fetch the schema by ID to deserialize. Compatibility modes gate registration: BACKWARD (the default) means readers on the new schema can decode old data — so you upgrade consumers first and may delete fields or add optional ones; FORWARD is the reverse, FULL is both, and the _TRANSITIVE variants check against all prior versions rather than just the latest. Picking the wrong mode is how one deploy breaks every downstream consumer.

KRaft (KIP-500) replaces ZooKeeper with a built-in Raft quorum of controller nodes that store all cluster metadata in an internal __cluster_metadata log, which brokers replay to stay current. This removes a second distributed system to operate, makes controller failover near-instant because standby controllers already hold the metadata, and raises the practical partition ceiling per cluster into the millions. ZooKeeper support was removed entirely in Kafka 4.0, so a 2026 answer should assume KRaft.