Debezium CDC Basics

Summary

Change Data Capture is the process of identifying and capturing changes made to a database and delivering those changes to downstream consumers in real-time. Unlike polling-based approaches, CDC captures every single change as it happens.

WHAT IS DEBEZIUM?

Debezium is an open-source distributed platform built on top of Kafka Connect. It provides source connectors for MySQL, PostgreSQL, MongoDB, SQL Server, Oracle, and other databases. These connectors read the database's transaction log to capture all changes with minimal impact on the source database.

Let's compare the JDBC Source Connector approach with Debezium CDC to understand the fundamental difference.

JDBC Source Connectors poll the database at regular intervals, introducing latency between when a change happens and when it's captured. Debezium reads the transaction log directly, capturing changes in real-time as they occur.

HOW DEBEZIUM WORKS

Debezium connectors read the database's transaction log, which is a low-level record of all changes made to the database. Different databases call this by different names: MySQL has the binlog, PostgreSQL has the WAL (Write-Ahead Log), and SQL Server has the transaction log.

Let's see how Debezium fits into the overall architecture.

Your application makes changes to the database as usual. The database writes these changes to its transaction log. Debezium reads from this log and produces change events to Kafka topics. The application doesn't need any modifications.

CHANGE EVENT STRUCTURE

Debezium produces rich change events that include both the before and after values of changed rows, along with metadata about the change.

Each change event contains:

before: The row's state before the change. Null for inserts.

after: The row's state after the change. Null for deletes.

source: Metadata including database name, table name, transaction ID, and log position.

op: The operation type - c for create (insert), u for update, d for delete, or r for read (initial snapshot).

ts_ms: Timestamp when the change occurred.

This rich structure allows downstream consumers to understand exactly what changed and react accordingly.

DEBEZIUM CHANGE EVENT EXAMPLE

The following shows a concrete example of a Debezium change event for an UPDATE operation.

1{
2  "before": {
3    "id": 1001,
4    "name": "Alice",
5    "email": "[email protected]",
6    "age": 30
7  },
8  "after": {
9    "id": 1001,
10    "name": "Alice",
11    "email": "[email protected]",
12    "age": 31
13  },
14  "source": {
15    "version": "2.0.0",
16    "connector": "mysql",
17    "name": "mysql-server-1",
18    "ts_ms": 1704067200000,
19    "db": "mydb",
20    "table": "users",
21    "server_id": 1,
22    "file": "mysql-bin.000003",
23    "pos": 12345
24  },
25  "op": "u",
26  "ts_ms": 1704067200050
27}

In this example, we can see that user 1001's email changed from [email protected] to [email protected], and their age changed from 30 to 31. The source metadata tells us exactly where in the MySQL binlog this change was recorded.

SUPPORTED DATABASES

Debezium provides connectors for multiple database systems, each with specific capabilities.

Each connector is tailored to work with the specific transaction log format and capabilities of its target database. MySQL and PostgreSQL connectors are the most mature and widely used.

DEBEZIUM CONNECTOR CONFIGURATION

The following shows a typical Debezium MySQL connector configuration.

1{
2  "name": "mysql-debezium-connector",
3  "config": {
4    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
5    "tasks.max": "1",
6    "database.hostname": "mysql-host",
7    "database.port": "3306",
8    "database.user": "debezium",
9    "database.password": "password",
10    "database.server.id": "184054",
11    "database.server.name": "mysql-server-1",
12    "database.include.list": "mydb",
13    "table.include.list": "mydb.users,mydb.orders",
14    "database.history.kafka.bootstrap.servers": "localhost:9092",
15    "database.history.kafka.topic": "schema-changes.mydb"
16  }
17}

Key configuration properties include:

connector.class: The Debezium MySQL connector class.

database.hostname, port, user, password: Database connection details.

database.server.id: A unique server ID for the binlog reader. Each connector instance needs a unique ID.

database.server.name: A logical name for this database server. This becomes the topic prefix.

database.include.list: Databases to capture changes from.

table.include.list: Specific tables to capture. You can also use table.exclude.list.

database.history.kafka.topic: Kafka topic where Debezium stores the database schema history. This allows the connector to handle schema changes over time.

INITIAL SNAPSHOT

When a Debezium connector first starts, it performs an initial snapshot of the database to capture the current state of all tables.

The initial snapshot ensures that all existing data is captured before streaming ongoing changes. Once the snapshot completes, the connector seamlessly transitions to streaming live changes from the transaction log.

BENEFITS OF DEBEZIUM CDC

Real-Time, Low Latency: Captures changes in milliseconds, not seconds or minutes. This enables truly real-time data pipelines.

Captures All Changes: Unlike polling, CDC captures every single insert, update, and delete, even if multiple changes happen between polls.

Minimal Database Impact: Reading transaction logs has minimal performance impact compared to running queries repeatedly.

No Schema Changes Required: You don't need to add timestamp columns or modify your database schema.

Complete Change History: The before and after values let you understand exactly what changed and build complete audit trails.

Let's recap what we learned about Debezium and Change Data Capture:

Key ideas

Debezium enables real-time Change Data Capture by reading database transaction logs instead of polling.

It captures every insert, update, and delete with millisecond-level latency and minimal database impact.

Change events include before and after states, operation type, and rich metadata for complete change tracking.

Debezium supports MySQL, PostgreSQL, MongoDB, SQL Server, Oracle, and other databases with dedicated connectors.

Initial snapshots capture existing data, then the connector seamlessly transitions to streaming live changes.

Use Debezium for real-time replication, event sourcing, cache invalidation, and building event-driven architectures.