Installing Kafka Connect

Summary

The good news is that Kafka Connect comes bundled with Apache Kafka, so if you've already installed Kafka, you already have Connect. If not, the installation process is nearly identical.

Note: This lesson builds on Kafka Fundamentals Chapter 2: Installing Kafka. If you need detailed Kafka installation steps, refer to that chapter.

When you install Kafka, you get several components. Kafka Connect is included as part of the distribution, along with scripts to run it.

The Kafka distribution includes connect-standalone.sh for standalone mode, connect-distributed.sh for distributed mode, and a few built-in connectors like FileStreamSource and FileStreamSink for testing.

Prerequisites

Before installing Kafka Connect, ensure you have:

Java: Kafka Connect requires Java 11 or later. Check your version:

1java -version

Kafka Cluster: You need a running Kafka cluster since Connect stores configuration and offsets in Kafka topics. If you don't have one, install Kafka first.

Sufficient Resources: Each Connect worker needs memory and CPU. Minimum 2GB RAM, but 4GB or more is recommended for production.

Installation Steps

Let's walk through the installation process:

Step 1: Download Kafka

If you haven't already, download Kafka from the Apache website:

1wget https://downloads.apache.org/kafka/3.6.0/kafka_2.13-3.6.0.tgz
2tar -xzf kafka_2.13-3.6.0.tgz
3cd kafka_2.13-3.6.0

Step 2: Verify Connect Scripts

Check that the Connect scripts are present:

1ls bin/connect-*.sh

You should see connect-standalone.sh and connect-distributed.sh.

Step 3: Check Built-in Connectors

Kafka includes two basic connectors for testing:

1ls libs/ | grep connect-file

You'll see connect-file JAR files for FileStreamSource and FileStreamSink connectors.

Here's the relevant directory structure after installation:

The bin directory has the startup scripts, libs contains the JAR files including connector implementations, and config has sample configuration files to get you started.

Configuration Files

Kafka provides sample configuration files in the config directory:

Worker Configuration

• connect-standalone.properties - for standalone mode
• connect-distributed.properties - for distributed mode

Connector Configuration

• connect-file-source.properties - example source connector
• connect-file-sink.properties - example sink connector

These sample files are a great starting point. You'll customize them based on your environment.

Verifying Installation

Let's verify the installation works. We'll do a quick test with standalone mode:

1# Start Zookeeper (if not using KRaft)
2bin/zookeeper-server-start.sh config/zookeeper.properties &
3
4# Start Kafka broker
5bin/kafka-server-start.sh config/server.properties &
6
7# Start Connect in standalone mode with file connectors
8bin/connect-standalone.sh \
9  config/connect-standalone.properties \
10  config/connect-file-source.properties \
11  config/connect-file-sink.properties

If you see log messages about starting connectors without errors, installation is successful.

Environment Variables

For production, you may want to set environment variables:

1# Kafka Connect home
2export KAFKA_HOME=/path/to/kafka
3
4# Add to PATH for convenience
5export PATH=$PATH:$KAFKA_HOME/bin
6
7# JVM heap size for Connect workers
8export KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"
9
10# Log directory
11export LOG_DIR=/var/log/kafka-connect

Add these to your .bashrc or .bash_profile for persistence.

Installing Kafka Connect is straightforward—it's included with Kafka. The real work is in configuration, which we'll cover