Connectors to Tasks

Summary

Understanding the relationship between connectors and tasks is crucial for scaling your Connect deployments and troubleshooting performance issues.

Let's start by understanding the fundamental relationship between a connector and its tasks. A connector is essentially a blueprint or controller that defines what work needs to be done, while tasks are the actual workers that execute that work.

Let's look more closely at what a connector actually does. The connector class has three main responsibilities.

First, the connector handles task configuration. It determines how many tasks can be created based on the source or sink system's characteristics. For example, a database connector might create one task per table, while a file connector might create one task per directory.

Second, the connector handles task distribution by generating individual configuration for each task. Each task config tells that specific task what portion of the work it should handle.

Third, the connector monitors task health and can request task restarts if failures are detected.

Key Concept: Tasks.Max Configuration

One of the most important configuration parameters for any connector is tasks.max. This parameter controls the maximum number of tasks the connector can create.

However, note that tasks.max is an upper limit. The connector may create fewer tasks if the workload doesn't support that level of parallelism. For instance, if you set tasks.max to 10 but only have 3 database tables to sync, you'll only get 3 tasks.

see what happens inside a task. Tasks follow a continuous poll-process-commit cycle.

For source connectors, tasks poll data from the external system, optionally transform it, send it to Kafka, and then commit their progress as offsets. For sink connectors, tasks consume from Kafka and write to the external system.

This continuous loop runs until the worker tells the task to stop, either due to reconfiguration, failure, or shutdown.

Task Parallelism Example

Let me give you a concrete example. Imagine you have a JDBC source connector reading from a MySQL database with 5 tables: users, orders, products, inventory, and audit_log.

If you configure tasks.max to 3, the connector might distribute the tables like this:

• Task 0: users, orders
• Task 1: products, inventory
• Task 2: audit_log

Each task independently polls its assigned tables and writes to Kafka. This parallelism significantly improves throughput compared to a single task handling all tables.

Task Rebalancing

One important behavior to understand is task rebalancing. When you add or remove workers in a distributed Connect cluster, or when you change the tasks.max configuration, Connect will rebalance tasks across available workers.

During a rebalance, tasks are stopped, redistributed, and restarted. This causes a brief interruption in data processing, but Connect handles it automatically. The connector uses stored offsets to resume from where each task left off, ensuring no data is lost.

Let's recap what we've learned. Connectors are controllers that manage tasks. Tasks are the actual workers that move data. The connector determines how to split work among tasks based on the source or sink system's characteristics and the tasks.max configuration.

Understanding this relationship helps you optimize your Connect deployments. More tasks mean more parallelism, but also more overhead. Finding the right balance depends on your workload and infrastructure.