Installing Kafka Without Zookeeper (KRaft Mode)

Overview

Kafka offers two installation methods:

• With Zookeeper: Traditional setup where Zookeeper handles metadata and leader elections
• Without Zookeeper (KRaft Mode): Introduced in Kafka 2.8, stable in 3.x versions

- Kafka manages its own metadata

- Simplifies architecture

- The future of Kafka

This guide covers Kafka 3.8.0 installation in KRaft mode for Linux, Mac, Windows, and Docker.

Prerequisites

Before installation, ensure you have:

• Java Development Kit (JDK) version 8 or higher (verify with java -version)
• Platform-specific tools:

- Windows: Git Bash or PowerShell

- Mac: Homebrew

- Linux: Package manager (apt or yum)

• Docker & Docker Compose (for containerized setup)

KRaft Mode on Linux

1. Download and Extract Kafka

1mkdir -p /opt/kafka
2wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
3tar -xvzf kafka_2.13-3.8.0.tgz -C /opt/kafka --strip-components=1

2. Create Log Directory

1mkdir /tmp/kraft-combined-logs

3. Configure server.properties

1cat <<EOL > /opt/kafka/config/kraft/server.properties
2process.roles=broker,controller
3node.id=1
4controller.quorum.voters=1@localhost:9093
5listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
6log.dirs=/tmp/kraft-combined-logs
7EOL

Configuration explained:

• process.roles=broker,controller: Node serves as both broker (handles client requests) and controller (manages metadata)
• node.id=1: Unique identifier for the Kafka node
• controller.quorum.voters=1@localhost:9093: Defines quorum voters for controller
• listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093:

- Broker listens for clients at port 9092

- Controller listens at port 9093

• log.dirs=/tmp/kraft-combined-logs: Directory for storing Kafka logs

4. Format Storage

1/opt/kafka/bin/kafka-storage.sh format -t $(uuidgen) -c /opt/kafka/config/kraft/server.properties

5. Start Kafka

1/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties

KRaft Mode on Mac

1. Install Kafka via Homebrew

1brew install kafka

2. Create Log Directory

1mkdir /tmp/kraft-combined-logs

3. Configure server.properties

1cat <<EOL > /usr/local/etc/kafka/server.properties
2process.roles=broker,controller
3node.id=1
4controller.quorum.voters=1@localhost:9093
5listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
6log.dirs=/tmp/kraft-combined-logs
7EOL

4. Format KRaft Metadata Storage

1/usr/local/bin/kafka-storage format -t $(uuidgen) -c /usr/local/etc/kafka/server.properties

5. Start Kafka

1kafka-server-start /usr/local/etc/kafka/server.properties

KRaft Mode on Windows

1. Create Directory and Extract

1mkdir C:\kafka
2tar -xvzf kafka_2.13-3.8.0.tgz -C C:\kafka

2. Create Log Directory

1mkdir C:\tmp\kraft-combined-logs

3. Configure server.properties

1cat <<EOL > C:\kafka\config\kraft\server.properties
2process.roles=broker,controller
3node.id=1
4controller.quorum.voters=1@localhost:9093
5listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
6log.dirs=C:\tmp\kraft-combined-logs
7EOL

4. Format KRaft Metadata Storage

1C:\kafka\bin\windows\kafka-storage.bat format -t $([guid]::NewGuid()) -c C:\kafka\config\kraft\server.properties

5. Start Kafka

1C:\kafka\bin\windows\kafka-server-start.bat C:\kafka\config\kraft\server.properties

Kafka in KRaft Mode - Docker

Step 1: Prepare Docker File

• Create a kafka directory in your workspace
• Create a docker-compose.yaml file in this directory

Step 2: Set Up Services

The docker-compose.yaml defines two services:

kafka-init service:

• Formats storage directory for KRaft mode
• Initializes metadata for Kafka
• Runs kafka-storage.sh format command
• Stores metadata in ./kafka-logs

kafka service:

• Configured as both broker and controller

Key environment variables:

• KAFKA_PROCESS_ROLES: Defines roles as broker and controller
• KAFKA_CONTROLLER_QUORUM_VOTERS: Identifies controller quorum node
• KAFKA_LISTENERS: Sets up client and controller listeners
• KAFKA_LOG_DIRS: Points to log storage directory

Step 3: Run Docker Compose

1docker-compose up -d

The -d flag runs services in detached mode, allowing them to run in the background while you execute other commands.