Transforming Keys, Values, and Headers

Summary

SMTs can transform keys, values, and headers separately. Understanding which to transform is crucial for correct behavior.

VALUE TRANSFORMS

Most SMTs transform the message value. Use $Value suffix:

1"transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value"

KEY TRANSFORMS

Transform message keys using $Key suffix:

1"transforms.extractId.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
2"transforms.extractId.field": "id"

HEADER TRANSFORMS

Add or modify headers:

1"transforms.addHeader.type": "org.apache.kafka.connect.transforms.InsertHeader",
2"transforms.addHeader.header": "source",
3"transforms.addHeader.value.literal": "database"

VALUETOKEY - COPY TO KEY

Copy a field from value to key for partitioning:

1"transforms.setKey.type": "org.apache.kafka.connect.transforms.ValueToKey",
2"transforms.setKey.fields": "user_id"

Key ideas

Use $Value for value transforms (most common).

Use $Key for key transforms.

Headers store metadata separately.

ValueToKey enables custom partitioning.

Chain transforms for complex operations.