Hands-On: FileStream Source

Summary

The FileStream Source Connector reads lines from a file and produces each line as a message to a Kafka topic. While it's not used much in production, it's excellent for testing and understanding connector fundamentals.

UNDERSTANDING FILESTREAM SOURCE

Let's start by understanding what the FileStream Source Connector does. It monitors a file on the local filesystem and reads new lines as they're appended to the file.

Here's how the FileStream Source Connector works. It reads from a file and produces each line as a message to a Kafka topic.

The connector watches the file and automatically produces new lines as messages. Each line becomes a single Kafka message with the line content as the message value.

CONNECTOR CONFIGURATION

look at how to configure a FileStream Source Connector. Connectors are configured using JSON. Here's a basic configuration:

1{
2  "name": "file-source-connector",
3  "config": {
4    "connector.class": "FileStreamSource",
5    "tasks.max": "1",
6    "file": "/tmp/input-file.txt",
7    "topic": "file-topic"
8  }
9}

name: A unique name for your connector instance. This is how you'll reference and manage the connector.

connector.class: Specifies which connector implementation to use. For FileStream Source, we use "FileStreamSource" or the full class name "org.apache.kafka.connect.file.FileStreamSource".

tasks.max: The maximum number of tasks to create. For FileStream Source, this should be 1 since you can't parallelize reading a single file.

file: The absolute path to the file to read from. If omitted, the connector reads from standard input.

topic: The Kafka topic to produce messages to.

HANDS-ON: RUNNING FILESTREAM SOURCE

walk through running a FileStream Source Connector in standalone mode.

Here's the workflow we'll follow to set up and run the connector.

Step 1: Create the Kafka Topic

First, create a topic to receive the messages:

1kafka-topics --create \
2  --bootstrap-server localhost:9092 \
3  --topic file-topic \
4  --partitions 3 \
5  --replication-factor 1

Step 2: Create an Input File

Create a file with some initial data:

1echo "First line" > /tmp/input-file.txt
2echo "Second line" >> /tmp/input-file.txt

Step 3: Configure the Connector

Create a connector configuration file at /tmp/filestream-source.properties:

1name=file-source-connector
2connector.class=org.apache.kafka.connect.file.FileStreamSource
3tasks.max=1
4file=/tmp/input-file.txt
5topic=file-topic

Step 4: Start Kafka Connect in Standalone Mode

Run Kafka Connect with the FileStream Source configuration:

1connect-standalone \
2  config/connect-standalone.properties \
3  /tmp/filestream-source.properties

Once the connector starts, you'll see log output showing it's reading from the file and producing to the topic. The connector will process the existing lines and then wait for new lines to be appended.

Step 5: Append New Data

In another terminal, append new lines to the file:

1echo "Third line" >> /tmp/input-file.txt
2echo "Fourth line" >> /tmp/input-file.txt

Step 6: Verify Messages

Consume from the topic to see the messages:

1kafka-console-consumer \
2  --bootstrap-server localhost:9092 \
3  --topic file-topic \
4  --from-beginning

You should see:

1First line
2Second line
3Third line
4Fourth line

UNDERSTANDING THE DATA FLOW

Let's visualize what happened during this hands-on exercise.

The connector continuously monitors the file for changes. When new lines are appended, it immediately reads and produces them to Kafka.

CONNECTOR OFFSET MANAGEMENT

The FileStream Source Connector maintains offsets to track which lines have been processed. This allows it to restart without re-processing data.

In standalone mode, offsets are stored in a local file. In distributed mode, they're stored in an internal Kafka topic. Either way, the connector can resume from where it left off.

Let's recap what we learned in this hands-on lesson:

Key ideas

The FileStream Source Connector reads lines from a file and produces them as Kafka messages. It's simple but perfect for learning and testing.

Configuration is done via properties or JSON, specifying the file path, topic name, and connector class.

The connector continuously monitors the file for new lines, producing them as they're appended.

Offsets are automatically managed, allowing the connector to restart without data loss or duplication.

While FileStream Source is basic, the concepts apply to all source connectors: configuration, continuous polling, offset management, and reliable delivery.