Plugin Paths and Connector Installation

Summary

So far, we've used the built-in FileStream connectors. But the real power of Kafka Connect comes from the ecosystem of third-party connectors for databases, cloud services, and more. To use these connectors, you need to install their JAR files and configure the plugin path.

Kafka Connect uses a plugin architecture. Workers load connector classes from JAR files in the plugin path at startup.

Each connector is organized in its own directory within the plugin path. The directory contains the connector JAR and all its dependencies. Connect scans these directories at startup and loads the connector classes.

Configuring Plugin Path

Set the plugin.path in your worker configuration:

1# Single plugin directory
2plugin.path=/opt/kafka/plugins
3
4# Multiple plugin directories (comma-separated)
5plugin.path=/opt/kafka/plugins,/usr/local/kafka-connectors,/home/user/custom-connectors

You can specify multiple directories separated by commas. Connect will scan all of them for connector plugins.

Plugin Directory Structure

The recommended directory structure isolates each connector:

1/opt/kafka/plugins/
2├── confluentinc-kafka-connect-jdbc/
3│   ├── kafka-connect-jdbc-10.7.4.jar
4│   ├── mysql-connector-java-8.0.30.jar
5│   ├── postgresql-42.5.1.jar
6│   └── sqlite-jdbc-3.40.0.0.jar
7├── confluentinc-kafka-connect-elasticsearch/
8│   ├── kafka-connect-elasticsearch-14.0.6.jar
9│   └── elasticsearch-rest-client-8.5.3.jar
10└── confluentinc-kafka-connect-s3/
11    ├── kafka-connect-s3-10.4.3.jar
12    ├── aws-java-sdk-s3-1.12.400.jar
13    └── aws-java-sdk-core-1.12.400.jar

Each connector has its own directory. This isolation prevents dependency conflicts between connectors. All JARs in a connector's directory are loaded together into an isolated classloader.

Installing a Connector

Let's walk through installing the JDBC connector as an example.

Step 1: Download the Connector

Download from Confluent Hub or the connector's GitHub releases:

1# Using Confluent Hub CLI
2confluent-hub install confluentinc/kafka-connect-jdbc:10.7.4
3
4# Or download manually
5wget https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.7.4/confluentinc-kafka-connect-jdbc-10.7.4.zip

Step 2: Extract to Plugin Path

1unzip confluentinc-kafka-connect-jdbc-10.7.4.zip -d /opt/kafka/plugins/

This creates /opt/kafka/plugins/confluentinc-kafka-connect-jdbc/ with all necessary files.

Step 3: Add Database Drivers

The JDBC connector needs database-specific drivers. Download and add them:

1# MySQL driver
2cd /opt/kafka/plugins/confluentinc-kafka-connect-jdbc/
3wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar
4
5# PostgreSQL driver
6wget https://jdbc.postgresql.org/download/postgresql-42.5.1.jar

Step 4: Restart Workers

Restart all Connect workers to load the new plugin:

1# Gracefully stop the worker
2kill -TERM <worker-pid>
3
4# Restart
5bin/connect-distributed.sh config/connect-distributed.properties

Here's what happens when a worker starts and loads plugins:

Workers scan the plugin path at startup, discover connector JARs, load them into isolated classloaders to prevent conflicts, and register them so they're available via REST API.

Verifying Installed Connectors

Check which connectors are available:

1curl http://localhost:8083/connector-plugins | jq

Response:

1[
2  {
3    "class": "org.apache.kafka.connect.file.FileStreamSinkConnector",
4    "type": "sink",
5    "version": "3.6.0"
6  },
7  {
8    "class": "org.apache.kafka.connect.file.FileStreamSourceConnector",
9    "type": "source",
10    "version": "3.6.0"
11  },
12  {
13    "class": "io.confluent.connect.jdbc.JdbcSinkConnector",
14    "type": "sink",
15    "version": "10.7.4"
16  },
17  {
18    "class": "io.confluent.connect.jdbc.JdbcSourceConnector",
19    "type": "source",
20    "version": "10.7.4"
21  }
22]

This endpoint shows all available connector plugins, their type (source or sink), and version. If your connector appears here, it's successfully installed.

Common Installation Issues

Issue 1: Connector Not Found

If the connector doesn't appear in the plugin list:

• Check that plugin.path is correctly set in worker config
• Verify the connector directory is in the plugin path
• Ensure the worker was restarted after adding the connector
• Check worker logs for classloading errors

Issue 2: Dependency Conflicts

If you get ClassNotFoundException or NoClassDefFoundError:

• Ensure all connector dependencies are in the same directory
• Check for conflicting versions of the same library
• Use isolated plugin directories to prevent conflicts

Issue 3: Permission Problems

If workers can't read plugin files:

1# Fix permissions
2chmod -R 755 /opt/kafka/plugins
3chown -R kafka-user:kafka-group /opt/kafka/plugins

Confluent Hub

The easiest way to install connectors is using Confluent Hub CLI:

1# Install Confluent Hub CLI
2curl -L https://cnfl.io/cli | sh -s -- -b /usr/local/bin
3
4# Install a connector
5confluent-hub install confluentinc/kafka-connect-jdbc:10.7.4 \
6  --component-dir /opt/kafka/plugins \
7  --worker-configs config/connect-distributed.properties

Confluent Hub automatically downloads the connector, extracts it to the right location, and even restarts workers if needed. It's much simpler than manual installation.

Best Practices

Organization

• One directory per connector in the plugin path
• Include all dependencies in the connector directory
• Use version numbers in directory names for clarity

Version Management

• Keep old versions when upgrading for easy rollback
• Document which connectors are installed and their versions
• Test connector upgrades in non-production first

Security

• Restrict write access to plugin directories
• Verify connector JARs from trusted sources
• Scan JARs for vulnerabilities before installing

In production, manage plugins carefully across all workers:

Installing connector plugins is straightforward: download, extract to the plugin path, add dependencies, and restart workers. With the right directory structure and plugin management, you can easily add new integrations to your Kafka Connect cluster.