What Are Single Message Transforms?

Summary

SMTs sit between connectors and Kafka, transforming data as it passes through. You can chain multiple SMTs together for complex transformations.

COMMON SMT USE CASES

Add or Remove Fields: Add timestamps, remove sensitive data.

Rename Fields: Map source field names to destination requirements.

Route Messages: Send to different topics based on content.

Mask Sensitive Data: Replace or hash PII fields.

Flatten Nested Structures: Convert nested JSON to flat records.

BUILT-IN SMTS

Kafka Connect includes many useful SMTs:

• InsertField: Add static fields or metadata
• ReplaceField: Rename or remove fields
• MaskField: Mask sensitive values
• ValueToKey / ExtractField: Manipulate keys
• TimestampRouter: Route to time-based topics
• RegexRouter: Route based on regex patterns
• Cast: Convert field types
• Filter: Drop specific records

CONFIGURATION EXAMPLE

1{
2  "name": "my-connector",
3  "config": {
4    "transforms": "insertTimestamp,maskEmail",
5    "transforms.insertTimestamp.type": "org.apache.kafka.connect.transforms.InsertField$Value",
6    "transforms.insertTimestamp.timestamp.field": "processed_at",
7    "transforms.maskEmail.type": "org.apache.kafka.connect.transforms.MaskField$Value",
8    "transforms.maskEmail.fields": "email",
9    "transforms.maskEmail.replacement": "***@***.com"
10  }
11}

Key ideas

SMTs provide lightweight data transformation without custom code.

Apply to source or sink connectors, transforming data in-flight.

Chain multiple SMTs for complex transformations.

Many built-in SMTs for common use cases.

Configure via connector properties, no code deployment needed.