Internal Topics

Summary

These internal topics are what make distributed Connect possible, enabling multiple workers to coordinate, share configuration, and track status across the cluster.

In distributed mode, Kafka Connect uses three internal topics to manage its state. Let's look at what each one does.

The connect-offsets topic stores the position of each connector task in its source or sink system. We covered this in detail in lesson 4.2.

The connect-configs topic stores all connector and task configurations. When you create or update a connector via the REST API, the config is written to this topic and distributed to all workers.

The connect-status topic tracks the runtime state of connectors and tasks, recording whether they're running, failed, paused, or stopped. This is what the REST API queries when you check connector status.

The connect-configs Topic

Let's dive deeper into each topic, starting with connect-configs. This topic uses log compaction to retain only the latest configuration for each connector.

1# Typical configuration
2config.storage.topic=connect-configs
3config.storage.replication.factor=3

When you submit a connector configuration via POST to the REST API, Connect writes it to the connect-configs topic. All workers read from this topic and apply the configuration, creating tasks as needed.

The topic is compacted, meaning older versions of a connector's config are deleted, keeping only the most recent version. The key is the connector name, and the value is the JSON configuration.

This approach enables dynamic reconfiguration without restarting workers. When you update a connector, the new config is written to the topic, all workers see it, and they stop old tasks and start new ones with the updated configuration.

Here's how configuration flows through the connect-configs topic:

The leader worker receives the configuration via REST API, validates it, and writes it to the connect-configs topic. All workers, including the leader, then read this config and update their local state.

This design ensures that all workers have a consistent view of which connectors should be running and how they're configured, even if the leader fails or the cluster is rebalanced.

The connect-status Topic

The connect-status topic tracks the runtime state of connectors and tasks. It records transitions between states like RUNNING, FAILED, PAUSED, and UNASSIGNED.

1# Typical configuration
2status.storage.topic=connect-status
3status.storage.replication.factor=3
4status.storage.partitions=5

Whenever a connector or task changes state, an entry is written to this topic. For example, when a task starts successfully, when it fails with an error, or when you pause a connector via the API.

The REST API reads from this topic to report current status. When you query GET /connectors/my-connector/status, Connect reads the latest status messages from this topic.

Like the configs topic, status is also compacted, retaining only the latest state for each connector and task.

Here's how status information flows:

Any state change, whether triggered by the user, a failure, or normal operation, results in a write to the status topic. This creates an audit trail of connector and task lifecycle events.

Monitoring tools can consume from this topic to track connector health over time, detect failures, and alert when connectors aren't running as expected.

The connect-offsets Topic

We covered the connect-offsets topic in lesson 4.2, but let's review its configuration here:

1# Typical configuration
2offset.storage.topic=connect-offsets
3offset.storage.replication.factor=3
4offset.storage.partitions=25

This topic stores the offsets for all source and sink connector tasks. More partitions allow for better parallelism when many tasks are writing offsets concurrently.

The topic uses log compaction with the key being a combination of connector name, task ID, and partition. The value contains the offset information specific to that task's work.

High replication factor is critical because losing offset data could cause duplicate processing or data loss when tasks restart.

Let's see how these three topics work together in a distributed Connect cluster:

All workers read configurations from connect-configs. Each task writes its offsets to connect-offsets and status updates to connect-status. This shared state is what enables the cluster to function as a coordinated unit.

If Worker 1 fails, Worker 2 can take over Task 1 by reading its last known offset from connect-offsets and resuming from there. The configuration is already available in connect-configs, and status is updated in connect-status.

Topic Configuration Best Practices

Set a high replication factor, typically 3 or more. These topics are critical infrastructure. Losing them means losing your cluster state.

Use sufficient partitions for the offsets topic, especially if you have many connectors. A good starting point is 25 partitions for moderate workloads, more for high-scale deployments.

Enable log compaction on all three topics. This is usually the default, but verify it's configured correctly.

Monitor the size and lag of these topics. If they grow excessively or consumers fall behind, it indicates a problem with your Connect cluster.

Inspecting Internal Topics

You can inspect these topics using standard Kafka tools:

1# List topics
2kafka-topics --bootstrap-server localhost:9092 --list | grep connect
3
4# Describe the offsets topic
5kafka-topics --bootstrap-server localhost:9092 \
6  --describe --topic connect-offsets
7
8# Read from status topic (be careful, could be large)
9kafka-console-consumer --bootstrap-server localhost:9092 \
10  --topic connect-status --from-beginning

Be cautious when reading from these topics in production, as they can be quite large. It's generally better to use the Connect REST API to inspect connector status and configuration rather than directly reading the internal topics.

Troubleshooting with Internal Topics

When troubleshooting Connect issues, the internal topics can provide valuable insights:

If a connector keeps restarting with the same error, check the connect-status topic to see the error history.

If tasks aren't resuming from the correct position after a restart, inspect the connect-offsets topic to verify offsets are being written correctly.

If configuration changes aren't taking effect across all workers, check that connect-configs is being replicated properly and all workers are consuming from it.

Internal Topic Deletion Warning

Never delete these internal topics while Connect is running. Doing so will cause catastrophic data loss. You'll lose all connector configurations, offsets, and status information.

If you need to reset a Connect cluster completely, stop all workers first, delete the topics, and restart the cluster. This should only be done in development environments or during disaster recovery.

Understanding Connect's internal topics gives you insight into how distributed Connect actually works under the hood. These topics enable the fault tolerance, scalability, and dynamic reconfiguration that make Connect a robust integration platform.

This completes Chapter 4 on Connect internals. You now understand how connectors create tasks, how offsets are stored, what converters do, how schemas work, and the role of internal topics.

In Chapter 5, we'll shift from internals to practical usage, exploring source connectors in depth. We'll see how to ingest data from files, databases, and change data capture systems into Kafka.