Source vs Sink Connectors

Summary

Kafka Connect connectors fall into two categories based on the direction of data flow. Source connectors bring data INTO Kafka from external systems, while sink connectors push data OUT of Kafka to external systems.

Let's visualize the difference. Source connectors read from external systems and produce to Kafka topics. Sink connectors consume from Kafka topics and write to external systems.

This separation of concerns makes it easy to build data pipelines. You might have a source connector reading from a database and producing to a topic, and then multiple sink connectors reading from that same topic to write to different destinations.

Let's dive deeper into how source connectors work. A source connector polls the external system for new data, converts it to Kafka records, and produces those records to topics.

The source connector has several responsibilities. First, it polls the source system—this could be querying a database table, reading new files, or calling an API. Second, it converts the data into Kafka records using a converter. Third, it produces those records to Kafka topics. Finally, it tracks its position using offsets, so if it restarts, it knows exactly where it left off.

Common source connector examples include:

• JDBC Source Connector: Polls database tables for new or updated rows
• Debezium CDC Connectors: Captures database change events in real-time
• FileStream Source: Reads lines from files
• HTTP Source: Polls REST APIs for data
• MQTT Source: Subscribes to IoT message streams

look at sink connectors. They consume records from Kafka topics, convert them to the target format, and write them to external systems.

The sink connector operates in the opposite direction. It consumes messages from one or more Kafka topics, converts them from Kafka's internal format to the target system's format, writes them to the destination, and commits offsets back to Kafka so it knows which messages have been successfully processed.

Common sink connector examples include:

• JDBC Sink Connector: Writes to database tables
• Elasticsearch Sink: Indexes documents for full-text search
• S3 Sink: Writes files to cloud object storage
• HDFS Sink: Writes to Hadoop distributed file system
• FileStream Sink: Writes lines to local files

Key Differences

Data Direction: Source connectors produce to Kafka, sink connectors consume from Kafka.

Offset Management: Source connectors track their own offsets in Connect's offset storage. Sink connectors use Kafka's consumer group offset mechanism.

Polling vs Consuming: Source connectors actively poll external systems on a schedule. Sink connectors passively consume from Kafka topics.

Idempotency: Source connectors often need to handle duplicate detection when restarting. Sink connectors can leverage Kafka's exactly-once semantics with proper configuration.

Understanding the distinction between source and sink connectors is fundamental to using Kafka Connect effectively. In practice, you'll often chain them together: a source connector brings data into Kafka, and one or more sink connectors distribute that data to various destinations.