Offset Storage

Summary

Without proper offset management, you could lose data or process duplicates. Let's understand how Connect handles offsets differently in standalone versus distributed mode.

First, let's understand what offsets mean in the context of Kafka Connect. Offsets are position markers that track where a connector has read up to in a source system or written up to in a sink system.

For source connectors, offsets might track database row IDs, file positions, or timestamps. For sink connectors, offsets track Kafka topic partitions and the last consumed offset from each partition.

When a task restarts after a failure or reconfiguration, it reads its last stored offset to resume from exactly where it left off. This is how Connect provides reliable, resumable data transfer.

Note: Reference to Kafka Fundamentals

The concept of offsets for sink connectors is similar to regular Kafka consumer offsets, which we covered in detail in Kafka Fundamentals lesson 4.5 Consumer Offsets. The key difference is where these offsets are stored, which we'll explore now.

In standalone mode, Connect stores offsets in a local file on the worker's filesystem. This is simple but has limitations.

In standalone mode, you configure a file path using the offset.storage.file.filename property. All tasks write their offsets to this single file in JSON format.

This approach works fine for development and testing, but it has serious drawbacks for production. The offset file is not replicated, so if the disk fails, you lose your offset history. Also, standalone mode doesn't support running multiple workers, so you can't scale horizontally.

Standalone Offset Configuration Example

Here's what a typical standalone worker configuration looks like for offset storage:

1# standalone.properties
2offset.storage.file.filename=/var/lib/kafka-connect/offsets
3offset.flush.interval.ms=10000

The offset.flush.interval.ms setting controls how frequently offsets are flushed to disk. Lower values mean more durability but more I/O overhead. The default is 60 seconds.

Distributed mode uses a much more robust approach. Instead of a local file, offsets are stored in an internal Kafka topic that is replicated across your Kafka cluster.

In distributed mode, all workers write their task offsets to a shared internal topic, typically named connect-offsets. This topic is configured with a high replication factor for durability.

Because offsets are stored in Kafka itself, they benefit from Kafka's reliability guarantees: replication, persistence, and log compaction. If a worker fails, another worker can take over its tasks and read the offsets from the shared topic.

Distributed Offset Configuration Example

Here's the configuration for distributed mode:

1# distributed.properties
2offset.storage.topic=connect-offsets
3offset.storage.replication.factor=3
4offset.storage.partitions=25
5offset.flush.interval.ms=10000

The offset.storage.topic defines the internal topic name. The replication factor should match or exceed your Kafka cluster's configuration for critical topics. More partitions allow for better parallelism when many connectors are running.

Offset Format and Compaction

The connect-offsets topic uses log compaction with a special key format. Each offset entry has a key that identifies the connector and task, and a value containing the offset data.

Compaction ensures that only the latest offset for each connector-task combination is retained, preventing the topic from growing indefinitely while still maintaining complete offset history.

Comparing Standalone vs Distributed Offset Storage

Let me summarize the key differences:

Standalone Mode

• File-based storage on local disk
• Simple to set up, good for development
• No high availability or fault tolerance
• Single worker only
• Risk of data loss if disk fails

Distributed Mode

• Kafka topic-based storage
• More complex setup, designed for production
• High availability through replication
• Multiple workers with automatic failover
• Durable and fault-tolerant

When to Use Each Approach

Use standalone mode for development, testing, or simple single-node deployments where high availability is not required. The file-based approach is easier to inspect and debug.

Use distributed mode for production environments where you need reliability, scalability, and fault tolerance. The extra complexity is worth it for mission-critical data pipelines.

Understanding offset storage is crucial for operating Connect reliably. The choice between file-based and topic-based storage impacts your system's durability, scalability, and operational complexity.