What Is a Sink Connector?

Summary

A sink connector is a component of Kafka Connect that consumes messages FROM Kafka topics and writes them TO external systems. Think of it as an automated data consumer that continuously reads from Kafka and delivers data to destinations like databases, search engines, cloud storage, or data warehouses.

WHAT IS A SINK CONNECTOR?

Sink connectors are the output side of Kafka Connect. They subscribe to Kafka topics, consume messages, and write them to external systems using the appropriate protocols and APIs.

Let's visualize how a sink connector works. It consumes messages from Kafka topics and writes them to external systems.

The sink connector runs within Kafka Connect, consuming messages from one or more Kafka topics and delivering them to the configured destination. It handles all the complexity of connecting to the external system, batching writes, and ensuring reliable delivery.

SINK CONNECTOR LIFECYCLE

A sink connector goes through several phases during its lifecycle. Let's see how it starts, consumes data, and handles writes.

The sink connector subscribes to topics, continuously polls for new records, and writes them to the external system in batches. After successful writes, it commits offsets back to Kafka to track progress, similar to how regular Kafka consumers work.

HOW SINK CONNECTORS TRACK PROGRESS

Like source connectors, sink connectors maintain offsets, but these are Kafka consumer offsets. They track which messages have been successfully written to the external system.

The sink connector commits offsets to Kafka only after data has been successfully written to the external system. This ensures at-least-once delivery semantics - if the connector fails before committing, the messages will be re-consumed and written again on restart.

COMMON TYPES OF SINK CONNECTORS

There are many types of sink connectors available for different destinations. Let me highlight the most common ones:

FileStream Sink Connector: Writes messages to files on the local filesystem. Simple but useful for testing and basic file export.

JDBC Sink Connector: Writes data to relational databases like MySQL, PostgreSQL, or SQL Server. Supports automatic table creation and schema evolution.

Elasticsearch Sink Connector: Indexes Kafka messages into Elasticsearch for full-text search and analytics. Perfect for building search and analytics pipelines.

S3 Sink Connector: Writes messages to AWS S3 in various formats (JSON, Avro, Parquet). Commonly used for data lakes and long-term storage.

Cloud Storage Connectors: Similar to S3 but for Google Cloud Storage or Azure Blob Storage.

BigQuery Sink: Loads data directly into Google BigQuery for analytics.

Snowflake Sink: Streams data into Snowflake data warehouse.

Let's visualize these different sink connector types and their destinations.

Each sink connector is specialized for its destination system, handling the specific protocols, APIs, and data formats required. This specialization means you get optimized, reliable writes without having to implement the integration yourself.

DATA TRANSFORMATION FLOW

Sink connectors don't just move data - they also handle data transformation and format conversion.

The sink connector deserializes Kafka messages using the configured converter (JSON, Avro, Protobuf, etc.), optionally applies Single Message Transforms for data manipulation, then converts the data to the format required by the destination system.

KEY DIFFERENCES: SOURCE VS SINK CONNECTORS

While source and sink connectors share many similarities, there are important differences in how they operate.

Source Connectors

• Pull data FROM external systems
• Produce messages TO Kafka
• Define their own offset format
• Poll external systems on intervals
• Handle initial snapshots

Sink Connectors

• Pull data FROM Kafka topics
• Write data TO external systems
• Use Kafka consumer offsets
• Continuously consume messages
• Handle backpressure from slow destinations

Let's recap what we learned about sink connectors:

Key ideas

Sink connectors consume messages FROM Kafka topics and write them TO external systems automatically.

They handle the complete lifecycle: subscribing to topics, consuming messages, transforming data, batching writes, and committing offsets.

Offset management uses standard Kafka consumer offsets, committed only after successful writes to ensure at-least-once delivery.

There's a rich ecosystem of sink connectors for databases, search engines, cloud storage, data warehouses, and more.

Sink connectors handle data format conversion and transformation, bridging Kafka's message formats with destination system requirements.

Like source connectors, sink connectors eliminate custom code, providing standardized, tested, and reliable integrations.