Standalone vs Distributed Mode

Summary

These deployment modes determine how workers coordinate, how fault tolerance works, and how you should configure your production environment.

Standalone mode is the simplest deployment. You run a single worker process on one machine. All connectors and tasks run in this single process.

In standalone mode, configuration and offsets are stored in local files. This makes it easy to get started and perfect for development, testing, or simple single-machine deployments. However, there's no fault tolerance—if the worker crashes, all connectors stop until you restart it.

Distributed mode is designed for production. Multiple workers form a cluster, and they coordinate using Kafka itself. Connectors and tasks are distributed across workers, and if a worker fails, tasks automatically rebalance to healthy workers.

Notice how distributed mode uses internal Kafka topics to store configurations, offsets, and status. All workers read from these topics to stay synchronized. When you submit a connector configuration via REST API to any worker, it's written to the config topic and all workers see it. Tasks are then distributed across the cluster.

Key Differences

Configuration Management

• Standalone: Local property files, edited manually
• Distributed: REST API, stored in Kafka topics, accessible from any worker

Offset Storage

• Standalone: Local file on disk
• Distributed: Kafka topic, replicated and durable

Scalability

• Standalone: Limited to one machine's resources
• Distributed: Add workers to scale horizontally

Fault Tolerance

• Standalone: None—worker failure stops all connectors
• Distributed: Automatic task rebalancing to healthy workers

Use Cases

• Standalone: Development, testing, simple single-machine pipelines
• Distributed: Production, high availability, large-scale deployments

Let's see what happens when a worker fails in distributed mode. The cluster detects the failure, and the remaining workers automatically redistribute the orphaned tasks.

This automatic rebalancing is why distributed mode is essential for production. Your data pipelines keep running even when individual workers fail. No manual intervention needed.

Starting Standalone Mode

To start in standalone mode, you use a command like this:

1connect-standalone.sh worker.properties connector1.properties connector2.properties

You provide a worker configuration file, followed by one or more connector configuration files. The worker starts immediately with those connectors.

Starting Distributed Mode

For distributed mode, you start workers with just the worker configuration:

1connect-distributed.sh worker.properties

Then you deploy connectors via REST API:

1curl -X POST http://localhost:8083/connectors \
2  -H "Content-Type: application/json" \
3  -d @connector-config.json

All workers in the cluster will see the new connector configuration and participate in running its tasks.

Choosing the right mode is important. For production systems where uptime matters, always use distributed mode. For local development and testing, standalone mode is simpler and faster to iterate with.