Converters

Summary

Choosing the right converter impacts your data format, schema evolution capabilities, and interoperability with other systems. Let's dive into the three most common converter types: JSON, Avro, and Protobuf.

First, let's understand where converters operate in the data pipeline. Converters sit between the connector tasks and Kafka, transforming data formats.

Source connectors read data in the source system's native format, then hand it to the converter as Connect's internal data representation. The converter serializes this into bytes that are written to Kafka.

On the sink side, the process reverses. The converter deserializes bytes from Kafka back into Connect's internal format, which the sink task then writes to the target system in its native format.

This decoupling means you can mix and match source formats with sink formats. For example, read from a database as Avro and write to Elasticsearch as JSON.

Kafka Connect supports several converter types. Let's look at the three most popular ones.

JSON converter produces human-readable text with optional schema information embedded in each message. It's easy to debug but produces larger messages.

Avro converter produces compact binary format with schema stored separately in a Schema Registry. It's space-efficient and provides strong schema evolution guarantees.

Protobuf converter also uses binary format with Schema Registry. It offers fast serialization and is popular in gRPC-based architectures.

JSON Converter Configuration

Let's start with JSON converter configuration. Here's a typical setup:

1# Connector configuration
2key.converter=org.apache.kafka.connect.json.JsonConverter
3value.converter=org.apache.kafka.connect.json.JsonConverter
4
5# Include schema in the JSON payload
6key.converter.schemas.enable=true
7value.converter.schemas.enable=true

The schemas.enable setting controls whether the schema is included with each message. When true, each JSON message contains both a schema section and a payload section. When false, only the payload is included, making messages smaller but losing schema information.

Here's an example of JSON with schema enabled:

1{
2  "schema": {
3    "type": "struct",
4    "fields": [
5      {"field": "id", "type": "int32"},
6      {"field": "name", "type": "string"}
7    ]
8  },
9  "payload": {
10    "id": 123,
11    "name": "John Doe"
12  }
13}

And here's the same message with schemas disabled:

1{
2  "id": 123,
3  "name": "John Doe"
4}

The second format is cleaner and smaller, but you lose type information and schema evolution capabilities.

Avro Converter Configuration

Avro is the most popular choice for production systems because it combines compact size with strong schema evolution support. Here's the configuration:

1# Connector configuration
2key.converter=io.confluent.connect.avro.AvroConverter
3value.converter=io.confluent.connect.avro.AvroConverter
4
5# Schema Registry URL
6key.converter.schema.registry.url=http://localhost:8081
7value.converter.schema.registry.url=http://localhost:8081

With Avro, schemas are stored in Schema Registry and referenced by ID in each message. This means the actual message contains just a small schema ID plus the binary-encoded payload, resulting in very compact messages.

Avro also provides excellent schema evolution capabilities with forward and backward compatibility, which we'll discuss more

Protobuf Converter Configuration

Protobuf is gaining popularity, especially in organizations already using Protocol Buffers for gRPC services. The configuration is similar to Avro:

1# Connector configuration
2key.converter=io.confluent.connect.protobuf.ProtobufConverter
3value.converter=io.confluent.connect.protobuf.ProtobufConverter
4
5# Schema Registry URL
6key.converter.schema.registry.url=http://localhost:8081
7value.converter.schema.registry.url=http://localhost:8081

Protobuf offers extremely fast serialization and deserialization, making it ideal for high-throughput scenarios. Like Avro, it uses Schema Registry for schema storage and supports schema evolution.

How do you choose which converter to use? Here's a decision tree to help.

If you need messages to be human-readable for debugging or you don't care about size, use JSON.

If you're already using Protobuf in your organization, stick with it for consistency.

If you need robust schema evolution or want compact message sizes, Avro is the best choice. It's the most widely used format in production Kafka deployments.

Key vs Value Converters

One important point: you can configure different converters for keys and values. For example:

1# Use String for keys, Avro for values
2key.converter=org.apache.kafka.connect.storage.StringConverter
3value.converter=io.confluent.connect.avro.AvroConverter
4value.converter.schema.registry.url=http://localhost:8081

This is common when keys are simple identifiers like user IDs or order numbers, and values contain complex structured data. Using StringConverter for keys avoids schema overhead for simple data.

Performance Considerations

JSON is slowest to serialize and deserialize, and produces the largest messages. Use it for low-throughput scenarios or when human readability is crucial.

Avro provides good balance between performance and features. Serialization is fast, and messages are compact.

Protobuf offers the fastest serialization, making it ideal for very high-throughput pipelines, though the difference from Avro is often marginal in practice.

Common Pitfalls

A common mistake is changing converters on an existing topic without considering compatibility. If you write messages with JSON converter and try to read with Avro converter, you'll get deserialization errors.

Always ensure your source and sink connectors use compatible converters, or plan a migration strategy if you need to change formats.

Converters are a fundamental building block in Kafka Connect. They determine your data format, schema management strategy, and message size. JSON offers simplicity and readability, Avro provides the best balance for production use, and Protobuf excels in high-performance scenarios.