What Is a Source Connector?

Summary

A source connector is a component of Kafka Connect that pulls data FROM external systems and writes it INTO Kafka topics. Think of it as an automated data producer that continuously monitors external systems and streams changes to Kafka.

Let's visualize how a source connector works. The source connector runs within Kafka Connect, reads data from an external system, and produces messages to Kafka topics.

The source connector doesn't just read data once. It continuously monitors the external system for new or changed data, ensuring that your Kafka topics stay up-to-date with the source system.

A source connector goes through several phases during its lifecycle. Let's see how it starts, polls for data, and handles changes.

HOW SOURCE CONNECTORS TRACK PROGRESS

One critical aspect of source connectors is offset management. Source connectors maintain offsets to track which data has already been read from the external system. This ensures that if the connector restarts, it doesn't re-process the same data.

Let's see how offset tracking works. The source connector stores offsets in Kafka's internal topics, allowing it to resume from the last position after a restart.

The offset mechanism varies depending on the connector. For a database connector, offsets might be database timestamps or auto-incrementing IDs. For a file connector, offsets might be file names and byte positions. The key is that offsets allow the connector to pick up exactly where it left off.

COMMON TYPES OF SOURCE CONNECTORS

There are many types of source connectors available. Let me highlight the most common ones:

FileStream Source Connector: Reads data from files on the local filesystem. Simple but useful for testing and basic file ingestion.

JDBC Source Connector: Connects to relational databases like MySQL, PostgreSQL, or Oracle. Can use timestamp columns or incrementing IDs to detect new or updated rows.

Debezium CDC Connectors: Capture change data from database transaction logs. Provides real-time streaming of all inserts, updates, and deletes with low latency.

HTTP Source Connector: Polls REST APIs at regular intervals and ingests the responses into Kafka.

Cloud Storage Connectors: Read data from cloud storage services like AWS S3, Google Cloud Storage, or Azure Blob Storage.

Let's visualize these different source connector types and what they connect to.

Now that you understand what a source connector is and how it works, let's recap the key points:

Key ideas

Source connectors pull data FROM external systems INTO Kafka topics. They run continuously, polling for new data and producing messages to Kafka.

They maintain offsets to track progress, ensuring exactly-once or at-least-once delivery semantics depending on configuration.

There's a rich ecosystem of source connectors for databases, files, APIs, cloud storage, messaging systems, and more.

Source connectors eliminate the need to write custom producer code, providing standardized, tested, and scalable solutions for data ingestion.