Kafka Consumer Group Assignment and Rebalance

Ever since Kafka 0.9, consumer group is managed by a group coordinator and a group leader.

Consumer Group Management

A group coordinator is one of the brokers in the cluster. It receives heartbeats from all consumers in the group.

A group leader is the first consumer of the group. It's responsible for partition assignment to all consumers in the group. It communicates with the group coordinator to get a list of live consumers and decides the subset of partitions for each consumer.

The decision is sent back to the group coordinator and in turn sent to the rest of consumers in the consumer group.

Rebalance

Each consumer in a consumer group takes several partitions of a topic. Rebalance is re-assignment of partitions ownership among consumers.

A rebalance is triggered in the following scenarios:

• A new consumer JOINs the group
• A consumer LEAVEs the group cleanly
• New partitions are added
• A consumer is considered DEAD by the group coordinator. This happens when the consumer cannot send heartbeat to the coordinator in time. It's typically due to long-running processing or crashing.