CentralMesh.io

AdSense Banner (728x90)

Kafka Connect

Intermediate

Build reliable, scalable data pipelines with Kafka Connect. Learn its architecture, deployment modes, REST API, connectors, converters, schemas, offsets, delivery guarantees, and Single Message Transforms through practical examples.

📚 35 lessons⏱️ 3 days

1.1 What Is Kafka Connect?

🎥

Understand the Kafka Connect framework and how it moves data between Kafka and external systems.

View Lesson →

1.2 Why Use Connect Instead of Custom Code?

🎥

Compare Kafka Connect with custom producers and consumers for common integration workloads.

View Lesson →

1.3 Source vs Sink Connectors

🎥

Learn how source connectors import data and sink connectors export Kafka records.

View Lesson →

1.4 Common Use Cases

🎥

Explore database integration, change data capture, search indexing, data lakes, and other patterns.

View Lesson →

1.5 Where Connect Fits in the Kafka Ecosystem

🎥

Choose between Kafka Connect, Kafka Streams, producers, and consumers for different jobs.

View Lesson →

2.1 Connect Workers, Tasks, and Connectors

🎥

Understand the worker, connector, and task hierarchy at the heart of Kafka Connect.

View Lesson →

2.2 Standalone vs Distributed Mode

🎥

Compare local standalone deployments with scalable, fault-tolerant distributed workers.

View Lesson →

2.3 Connect REST API

🎥

Use Kafka Connect REST endpoints to create, inspect, update, pause, and delete connectors.

View Lesson →

2.4 Parallelism and Fault Tolerance

Learn how tasks scale work and how distributed workers recover from failures.

View Lesson →

3.1 Installing Kafka Connect

Install Kafka Connect, verify its scripts, and prepare a working local environment.

View Lesson →

3.2 Standalone Worker Setup

Configure and run a standalone worker with connector property files.

View Lesson →

3.3 Distributed Worker Setup

Configure a production-oriented distributed worker group and its internal topics.

View Lesson →

3.4 Plugin Paths and Connector Installation

Install third-party connector JARs and manage isolated plugin paths safely.

View Lesson →

3.5 Managing Connect with the REST API

Practice the complete connector management lifecycle through REST calls.

View Lesson →

4.1 Connectors to Tasks

See how connectors generate task configurations and distribute work.

View Lesson →

4.2 Offset Storage

Understand how source and sink progress is recorded, committed, and recovered.

View Lesson →

4.3 Converters

Configure JSON, Avro, Protobuf, and primitive converters for keys and values.

View Lesson →

4.4 Schemas in Connect

Learn how Connect data and schemas support validation and controlled evolution.

View Lesson →

4.5 Internal Topics

Understand the config, offset, and status topics used by distributed workers.

View Lesson →

5.1 What Is a Source Connector?

Follow the source connector lifecycle from external polling to Kafka records.

View Lesson →

5.2 Hands-On: FileStream Source

Build a working pipeline that streams new lines from a file into Kafka.

View Lesson →

5.3 JDBC Source Connector

Stream relational database rows using bulk, incrementing, timestamp, and combined modes.

View Lesson →

5.4 Debezium CDC Basics

Capture database changes from transaction logs with Debezium connectors.

View Lesson →

5.5 Source Offsets and Restart Behavior

Understand offset persistence, restarts, reprocessing, and recovery for source connectors.

View Lesson →

6.1 What Is a Sink Connector?

Follow records from Kafka topics through sink tasks into external systems.

View Lesson →

6.2 Hands-On: FileStream Sink

Build a working sink connector that writes Kafka records to a local file.

View Lesson →

6.3 JDBC Sink Connector

Write Kafka records into relational tables with inserts, upserts, deletes, and schema evolution.

View Lesson →

6.4 Elasticsearch Sink Connector

Index Kafka records in Elasticsearch for real-time search and analytics.

View Lesson →

6.5 S3 Sink Connector

Land partitioned Kafka data in Amazon S3 for data lake and analytics workloads.

View Lesson →

6.6 Delivery Guarantees

Reason about at-most-once, at-least-once, exactly-once, retries, and idempotency.

View Lesson →

7.1 What Are Single Message Transforms?

Use lightweight transformations to reshape individual records inside connectors.

View Lesson →

7.2 Common Single Message Transforms

Apply built-in transforms for routing, filtering, field changes, timestamps, and keys.

View Lesson →

7.3 Transforming Keys, Values, and Headers

Target the correct part of a Connect record and chain transforms predictably.

View Lesson →

7.4 Hands-On: Single Message Transforms

Build and test a practical multi-transform connector pipeline.

View Lesson →

7.5 Writing Custom Single Message Transforms

Implement, package, configure, and test a custom Java transformation.

View Lesson →