Kafka Configuration

Overview

Kafka's behavior is controlled through several key configuration files that work differently depending on whether you're using Zookeeper or running in KRaft mode. Understanding these configuration files is crucial for proper Kafka deployment and operation.

Main Configuration Files

Kafka has three primary configuration files:

• server.properties: Configures the Kafka broker itself
• zookeeper.properties: Used when running Kafka with Zookeeper for cluster management
• kraft.properties: Used when running Kafka without Zookeeper in KRaft mode

Configuration with Zookeeper

When Kafka uses Zookeeper for coordination, the server.properties file controls most of the broker's behavior.

Key Settings in server.properties

• zookeeper.connect: Specifies where Kafka can find the Zookeeper instance that manages the cluster
• log.dirs: Defines where Kafka stores its log files
• listeners: Specifies the network addresses Kafka binds to for receiving client connections

Zookeeper Properties

The zookeeper.properties file controls Zookeeper's behavior and is essential for Kafka's coordination tasks.

Key settings include:

• dataDir: Directory where Zookeeper stores its data
• clientPort: Port where Zookeeper listens for client connections (typically 2181)

This configuration creates a dependency structure where:

• Kafka Broker connects to Zookeeper
• Kafka Broker manages log directories
• Kafka Broker configures listeners for client connections
• Zookeeper manages its own data directory and client port

Configuration in KRaft Mode

When using KRaft mode, Zookeeper is eliminated and Kafka handles cluster management internally.

Key Settings in kraft.properties

The kraft.properties file is similar to server.properties but includes KRaft-specific settings:

• process.roles=controller,broker: Defines the roles this Kafka instance plays (can be broker, controller, or both)
• controller.quorum.voters: Defines the controllers participating in the quorum for metadata management
• log.dirs: Same as before - defines where log files are stored
• listeners: Defines the network addresses Kafka binds to for both client and controller communication

KRaft Architecture

In KRaft mode:

• Kafka Broker includes an internal Controller component
• The Controller manages metadata without external dependencies
• Controller Quorum Voters coordinate among themselves
• Log directories and listeners are configured similarly to Zookeeper mode

Key Differences

The main architectural difference is that KRaft eliminates the external dependency on Zookeeper by embedding the coordination logic directly into Kafka. This simplifies the overall architecture and reduces operational overhead while maintaining the same core functionality.

With Zookeeper

• External service required
• Separate configuration and management
• Additional infrastructure to maintain

With KRaft

• Self-contained coordination
• Simpler operational model
• Reduced infrastructure complexity