Parallelism and Fault Tolerance

Summary

Parallelism in Kafka Connect is achieved through tasks, while fault tolerance comes from distributed mode and Kafka's underlying replication. Understanding these concepts helps you design resilient, high-throughput data pipelines.

Note: This lesson builds on concepts from Kafka Fundamentals 3.5 Cluster Scaling, where we learned about Kafka's partitioning and replication.

Parallelism in Kafka Connect is straightforward: more tasks means more parallelism. Each task is an independent unit that can run on any worker in the cluster.

The connector decides how to split work across tasks. For a database connector, you might have one task per table. For a file connector, one task per file or directory. Each task operates independently, reading or writing data in parallel.

Controlling Parallelism

You control parallelism with the tasks.max configuration parameter:

1{
2  "name": "jdbc-source",
3  "config": {
4    "connector.class": "JdbcSourceConnector",
5    "tasks.max": "5",
6    "connection.url": "jdbc:postgresql://localhost:5432/inventory",
7    "mode": "incrementing",
8    "incrementing.column.name": "id",
9    "topic.prefix": "inventory-"
10  }
11}

Setting tasks.max to 5 tells the connector it can create up to 5 tasks. The actual number depends on the work available—if you're reading from 3 database tables, you'll get 3 tasks, not 5.

Parallelism Best Practices

• Start with tasks.max equal to the number of source partitions or tables
• For sink connectors, match the number of Kafka topic partitions
• Don't set tasks.max higher than the number of workers
• Monitor throughput and adjust based on bottlenecks

Fault tolerance in Kafka Connect comes from distributed mode. When a worker fails, the cluster automatically rebalances tasks to healthy workers. No data is lost because offsets are stored in replicated Kafka topics.

When Worker 2 fails, its task (Task 2) is automatically reassigned to Worker 1. The task picks up exactly where it left off because offset information is stored in Kafka's offset topic, which is replicated. This is similar to how Kafka consumers use consumer group coordination for fault tolerance.

Offset Storage and Recovery

In distributed mode, offsets are stored in the internal Kafka topic named connect-offsets. This topic is replicated across multiple Kafka brokers, making it highly available.

For Source Connectors

Source connectors track their position in the source system. For example, a database connector stores the last ID it read, or the last timestamp. When a task restarts, it reads its offset from the connect-offsets topic and continues from that position.

For Sink Connectors

Sink connectors use standard Kafka consumer offsets to track which messages they've written to the external system. These offsets are also stored in Kafka, providing exactly-once or at-least-once delivery guarantees depending on configuration.

The following shows the rebalancing process in detail. When workers join or leave the cluster, Connect triggers a rebalance to redistribute tasks fairly across available workers.

The rebalancing process is automatic and fast, typically completing in seconds. This is why distributed mode is essential for production—your data pipelines self-heal without manual intervention.

Scaling Connect Clusters

Similar to Kafka Fundamentals 3.5 Cluster Scaling, you can scale Kafka Connect clusters by adding or removing workers:

Scaling Up (Adding Workers)

1. Start a new worker with the same group.id configuration
2. The new worker joins the cluster
3. Rebalance automatically redistributes tasks
4. Higher total throughput achieved

Scaling Down (Removing Workers)

1. Gracefully stop a worker
2. Cluster detects the departure
3. Remaining workers rebalance and absorb the tasks
4. No data loss due to offset storage in Kafka

Best Practices

For High Availability

• Run at least 3 workers in distributed mode
• Ensure worker machines are in different availability zones
• Configure connect-offsets, connect-configs, and connect-status topics with replication factor of 3

For Performance

• Set tasks.max based on available parallelism in your source/sink
• Don't over-parallelize—too many tasks can cause overhead
• Monitor task lag and adjust task count if needed
• Consider worker CPU and memory when scaling

For Reliability

• Use distributed mode in production
• Enable automatic restarts for transient failures
• Monitor connector and task status via REST API
• Set up alerts for failed tasks

Parallelism and fault tolerance are what make Kafka Connect production-ready. By leveraging tasks for parallelism and distributed mode for fault tolerance, you can build data pipelines that scale and self-heal.