Connect Workers, Tasks, and Connectors

Summary

Kafka Connect has three main components: Workers, Connectors, and Tasks. These components work together to move data reliably between Kafka and external systems.

Let's start by looking at how these three components relate to each other. At the top level, you have the Connect Worker process, which manages Connectors, and each Connector coordinates multiple Tasks.

Let's break down each component. The Worker is a JVM process that executes connectors and tasks. You can run one or more workers depending on your deployment mode. The Worker handles the runtime environment and resource management.

The Connector is a logical job that manages data movement for a specific integration. For example, a JDBC connector for reading from MySQL, or an S3 connector for writing to cloud storage. The Connector decides how to split work into tasks.

Tasks are where the actual work happens. Each Task is responsible for copying a subset of data. For example, if you're reading from a database with 10 tables, you might have 10 tasks, one per table. Tasks are the unit of parallelism in Kafka Connect.

Here's how data flows through these components. External data comes in, Tasks process it, Connectors coordinate the tasks, and Workers manage the entire operation while producing to or consuming from Kafka.

This separation of responsibilities is powerful. The Worker provides the infrastructure—it handles configuration storage, REST APIs, and cluster coordination. The Connector provides the business logic for a specific integration—it knows how to connect to a particular system and how to split work. Tasks do the heavy lifting—they actually read or write data.

Key Responsibilities

Let me detail the responsibilities of each component:

Worker Responsibilities

• Running the JVM process and allocating resources
• Providing REST API for connector management
• Storing connector and task configurations
• Monitoring connector and task health
• Rebalancing tasks when workers join or leave
• Managing offset storage for source connectors
• Coordinating with other workers in distributed mode

Connector Responsibilities

• Determining how many tasks are needed
• Splitting work across tasks
• Deciding task configuration
• Monitoring task status
• Requesting task reconfiguration when needed

Task Responsibilities

• Actually reading from or writing to external systems
• Converting data to and from Kafka's format
• Tracking progress via offsets
• Handling errors for individual records
• Reporting metrics and status

The following shows what happens when you deploy a connector. You send configuration to the Worker REST API, the Worker creates the Connector, the Connector determines how to split work and creates Task configs, and the Worker distributes Tasks across available Workers.

This lifecycle shows the coordination between components. When you create a connector, you're really just providing configuration. The Connector logic determines how to split work, and the Worker handles the actual task distribution and execution.

Understanding this architecture helps you make better decisions about configuration. For example, if you need more parallelism, you increase the number of tasks. If you need fault tolerance, you add more workers. If you need a new integration, you deploy a new connector.