Kafka Connect
IntermediateBuild reliable, scalable data pipelines with Kafka Connect. Learn its architecture, deployment modes, REST API, connectors, converters, schemas, offsets, delivery guarantees, and Single Message Transforms through practical examples.
1.1 What Is Kafka Connect?
🎥Understand the Kafka Connect framework and how it moves data between Kafka and external systems.
1.2 Why Use Connect Instead of Custom Code?
🎥Compare Kafka Connect with custom producers and consumers for common integration workloads.
1.3 Source vs Sink Connectors
🎥Learn how source connectors import data and sink connectors export Kafka records.
1.4 Common Use Cases
🎥Explore database integration, change data capture, search indexing, data lakes, and other patterns.
1.5 Where Connect Fits in the Kafka Ecosystem
🎥Choose between Kafka Connect, Kafka Streams, producers, and consumers for different jobs.
2.1 Connect Workers, Tasks, and Connectors
🎥Understand the worker, connector, and task hierarchy at the heart of Kafka Connect.
2.2 Standalone vs Distributed Mode
🎥Compare local standalone deployments with scalable, fault-tolerant distributed workers.
2.3 Connect REST API
🎥Use Kafka Connect REST endpoints to create, inspect, update, pause, and delete connectors.
2.4 Parallelism and Fault Tolerance
Learn how tasks scale work and how distributed workers recover from failures.
3.1 Installing Kafka Connect
Install Kafka Connect, verify its scripts, and prepare a working local environment.
3.2 Standalone Worker Setup
Configure and run a standalone worker with connector property files.
3.3 Distributed Worker Setup
Configure a production-oriented distributed worker group and its internal topics.
3.4 Plugin Paths and Connector Installation
Install third-party connector JARs and manage isolated plugin paths safely.
3.5 Managing Connect with the REST API
Practice the complete connector management lifecycle through REST calls.
4.1 Connectors to Tasks
See how connectors generate task configurations and distribute work.
4.2 Offset Storage
Understand how source and sink progress is recorded, committed, and recovered.
4.3 Converters
Configure JSON, Avro, Protobuf, and primitive converters for keys and values.
4.4 Schemas in Connect
Learn how Connect data and schemas support validation and controlled evolution.
4.5 Internal Topics
Understand the config, offset, and status topics used by distributed workers.
5.1 What Is a Source Connector?
Follow the source connector lifecycle from external polling to Kafka records.
5.2 Hands-On: FileStream Source
Build a working pipeline that streams new lines from a file into Kafka.
5.3 JDBC Source Connector
Stream relational database rows using bulk, incrementing, timestamp, and combined modes.
5.4 Debezium CDC Basics
Capture database changes from transaction logs with Debezium connectors.
5.5 Source Offsets and Restart Behavior
Understand offset persistence, restarts, reprocessing, and recovery for source connectors.
6.1 What Is a Sink Connector?
Follow records from Kafka topics through sink tasks into external systems.
6.2 Hands-On: FileStream Sink
Build a working sink connector that writes Kafka records to a local file.
6.3 JDBC Sink Connector
Write Kafka records into relational tables with inserts, upserts, deletes, and schema evolution.
6.4 Elasticsearch Sink Connector
Index Kafka records in Elasticsearch for real-time search and analytics.
6.5 S3 Sink Connector
Land partitioned Kafka data in Amazon S3 for data lake and analytics workloads.
6.6 Delivery Guarantees
Reason about at-most-once, at-least-once, exactly-once, retries, and idempotency.
7.1 What Are Single Message Transforms?
Use lightweight transformations to reshape individual records inside connectors.
7.2 Common Single Message Transforms
Apply built-in transforms for routing, filtering, field changes, timestamps, and keys.
7.3 Transforming Keys, Values, and Headers
Target the correct part of a Connect record and chain transforms predictably.
7.4 Hands-On: Single Message Transforms
Build and test a practical multi-transform connector pipeline.
7.5 Writing Custom Single Message Transforms
Implement, package, configure, and test a custom Java transformation.