Delivery Guarantees

Summary

DELIVERY SEMANTICS

Sink connectors provide different delivery guarantees depending on configuration:

At-Least-Once: Messages may be delivered more than once. Default for most connectors.

At-Most-Once: Messages may be lost but never duplicated. Rarely used.

Exactly-Once: Each message delivered exactly once. Requires idempotent writes.

AT-LEAST-ONCE DELIVERY

Standard sink connector behavior: offset committed AFTER successful write.

ACHIEVING EXACTLY-ONCE

Combine idempotent writes with proper offset management:

1. Use primary keys (JDBC UPSERT mode)
2. Use Kafka keys as document IDs (Elasticsearch)
3. Rely on destination deduplication

OFFSET COMMIT STRATEGIES

Configuration affects reliability:

1{
2  "offset.flush.interval.ms": "60000"
3}

Frequent commits: Less reprocessing on failure, more overhead.

Infrequent commits: Better performance, more duplicates on failure.

Key ideas

Sink connectors default to at-least-once delivery.

Exactly-once requires idempotent writes at the destination.

Use primary keys, UPSERT mode, or destination deduplication.

Offset commit frequency balances reliability vs performance.

Design downstream systems to handle potential duplicates gracefully.