Standalone Worker Setup

Summary

Now that Kafka Connect is installed, let's configure and run it in standalone mode. Standalone mode is perfect for learning, development, and simple single-machine deployments.

In standalone mode, you configure everything through property files and start the worker with a single command. All configuration and offset data is stored locally on disk.

Remember from earlier lessons, standalone mode runs one worker process with all connectors and tasks in that single JVM.

The worker config defines how the worker operates, while connector configs define what data to move. The worker stores offset information in a local file so it can resume after restarts.

Worker Configuration

The following shows the key settings in connect-standalone.properties:

1# Kafka broker connection
2bootstrap.servers=localhost:9092
3
4# Converter settings for key and value
5key.converter=org.apache.kafka.connect.json.JsonConverter
6value.converter=org.apache.kafka.connect.json.JsonConverter
7
8# Whether to include schemas in the JSON
9key.converter.schemas.enable=false
10value.converter.schemas.enable=false
11
12# Local file to store offsets
13offset.storage.file.filename=/tmp/connect.offsets
14
15# Flush interval for offsets (milliseconds)
16offset.flush.interval.ms=10000
17
18# Plugin path for additional connectors
19plugin.path=/opt/kafka/plugins

The bootstrap.servers setting tells Connect where to find your Kafka cluster. The converter settings determine how data is serialized when writing to Kafka. The offset storage file tracks progress for source connectors.

Creating a File Source Connector

Create a simple file source connector that reads from a text file and writes each line to a Kafka topic.

Create a file named file-source.properties:

1name=local-file-source
2connector.class=FileStreamSource
3tasks.max=1
4file=/tmp/test-source.txt
5topic=connect-test

This connector reads from /tmp/test-source.txt and produces each line to the connect-test topic. It's simple but great for learning and testing.

Creating a File Sink Connector

create a sink connector that reads from Kafka and writes to a file.

Create file-sink.properties:

1name=local-file-sink
2connector.class=FileStreamSink
3tasks.max=1
4file=/tmp/test-sink.txt
5topics=connect-test

This connector consumes from the connect-test topic and writes each message to /tmp/test-sink.txt. Together with the source connector, this creates a simple pipeline: file → Kafka → file.

Here's the complete data flow we're setting up:

Data flows from the source file, through the source connector to Kafka, then from Kafka through the sink connector to the destination file. The worker manages both connectors in a single process.

Starting Standalone Worker

start the worker. First, ensure Kafka is running:

1# Start Zookeeper (if needed)
2bin/zookeeper-server-start.sh config/zookeeper.properties &
3
4# Start Kafka broker
5bin/kafka-server-start.sh config/server.properties &

Then start Connect in standalone mode:

1bin/connect-standalone.sh \
2  config/connect-standalone.properties \
3  config/file-source.properties \
4  config/file-sink.properties

Notice the command structure: the standalone script, followed by the worker config, followed by one or more connector configs. Connect will start and immediately begin running those connectors.

Testing the Pipeline

Let's test our pipeline by writing data to the source file:

1# Create test data
2echo "Hello Kafka Connect" >> /tmp/test-source.txt
3echo "This is line 2" >> /tmp/test-source.txt
4echo "And line 3" >> /tmp/test-source.txt
5
6# Check that data appears in the sink file
7cat /tmp/test-sink.txt

You should see the same lines appear in test-sink.txt after a moment.

Monitoring Standalone Worker

Check the console output for status information:

1# Look for these messages:
2[INFO] Connector local-file-source config updated
3[INFO] Task local-file-source-0 config updated
4[INFO] Starting task local-file-source-0
5[INFO] WorkerSourceTask{id=local-file-source-0} Source task finished initialization and start

These log messages confirm that connectors and tasks started successfully.

Common Configuration Options

Here are additional useful worker configuration options:

1# REST API configuration (even in standalone)
2rest.port=8083
3rest.host.name=localhost
4
5# Consumer and producer overrides
6consumer.max.poll.records=500
7producer.batch.size=16384
8producer.linger.ms=10
9
10# Task shutdown timeout
11task.shutdown.graceful.timeout.ms=5000
12
13# Connector client overrides prefix
14connector.client.config.override.policy=All

Stopping Standalone Worker

To stop the worker gracefully, use Ctrl+C in the terminal where it's running. The worker will:

1. Stop all tasks
2. Flush offsets to disk
3. Close connections
4. Shut down cleanly

The offset file preserves progress, so restarting will resume from where it stopped.

Standalone mode is great for getting started with Kafka Connect. It's simple to configure, easy to debug, and perfect for development environments. However, for production, you'll want distributed mode for fault tolerance and scalability, which we'll cover