Kafka Connect for CDC: Distributed Mode, SMTs, and Production Configuration

Most Kafka Connect deployments start in standalone mode because the docs make it look simple. You write a properties file, point at the source, and rows start flowing into Kafka. Then a worker restarts, the local offset file is gone, and the connector initiates a full re-snapshot of a 500-million-row table at 2am. That’s when teams discover that Kafka Connect configuration decisions are harder to change after they’ve affected production than before.

This guide covers the decisions that matter for CDC workloads specifically: mode, offset storage, transforms, delete handling, and scaling.

Standalone vs Distributed Mode: Which to Run in Production

Kafka Connect runs in two modes, and the difference comes down to where connector offsets are stored.

Standalone mode runs a single worker process. Connector offsets live in a file on that worker’s local disk. Configuration is simple: you pass two properties files (one for the worker, one for the connector) and start the process. For local development or a one-off data migration with no restart requirements, standalone is fine.

The problem with standalone for production CDC: the offset file is local. If the worker host is replaced (scaling event, cloud instance recycled, OS update), the file is gone. The connector sees no existing offset and starts a fresh snapshot of the source database: on a large table, that can mean hours of database load and a flood of backfill events hitting Kafka topics your consumers aren’t expecting.

Distributed mode stores offsets in a Kafka topic (connect-offsets by default). All worker processes in the cluster read and write to that topic, so any worker can pick up where another left off. A connector in distributed mode survives worker restarts, worker replacements, and horizontal scaling without losing its position.

Distributed mode requires at least two configuration topics: connect-offsets (offset storage), connect-configs (connector configuration state), and connect-status (task status). These topics need to exist before the first worker starts, with appropriate replication factors for your Kafka cluster’s durability requirements.

# Worker properties excerpt (distributed mode)
group.id=cdc-connect-cluster
config.storage.topic=connect-configs
offset.storage.topic=connect-offsets
status.storage.topic=connect-status
config.storage.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3

Two workers is the minimum for fault tolerance. One worker still runs the connector task after the other fails; Kafka Connect rebalances task assignment automatically.

Offset Management and Connector Restart Behaviour

In distributed mode, the connector’s offset is a key-value entry in the connect-offsets topic. The key identifies the connector and the specific source partition (e.g. a database + table combination), and the value is the last committed position.

On a normal restart, the worker reads its connector’s offset from the topic and resumes from that position. The source database sees no new connection beyond the usual reconnect.

On a connector reset (manual or after a configuration change that affects the source scope), the offset needs to be cleared. The procedure:

1. Stop the connector via the REST API (DELETE /connectors/{name} or a POST to pause + reset, depending on your connector version).
2. Produce a null-payload message to connect-offsets with the exact key used by the connector’s offset entry. This tombstone tells Kafka log compaction to drop the entry.
3. Restart the connector. With no offset found, it starts a fresh snapshot.

The “exact key” step is the tricky part. For Debezium-style connectors, the offset key is a JSON structure encoding the connector name and source partition. You can read the current key by consuming connect-offsets and finding the entry for your connector name. Don’t guess at the key format: an incorrect key doesn’t clear the offset, and the connector will silently continue from the old position.

Single-Message Transforms for CDC Event Routing

Single-message transforms (SMTs) run on each Kafka Connect event before it’s written to the destination topic. For CDC workloads, three are especially useful.

Topic routing redirects events from the connector’s default topic assignment to a specific topic based on the event’s content. For CDC, this is how you split a multi-table capture into per-table topics:

"transforms": "route",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "(.*)server1\\.(.*)\\.(.*)",
"transforms.route.replacement": "cdc.$2.$3"

Field masking removes or redacts sensitive fields before the event leaves the connector. This is the correct place to strip PII that shouldn’t reach the target topic. MaskField replaces field values with a constant (null, zero, an empty string) without removing the field from the schema; ReplaceField removes the field entirely.

Flatten converts nested change-event structures into flat records. Useful when the destination consumer expects a simple key-value record rather than a CDC envelope with before, after and source sections.

SMT execution order is critical and is a common source of silent bugs. Transforms run in the order they’re listed under transforms. A transform that reads a field (Router reading table) will produce unexpected results if it runs after a transform that already removed or renamed that field (ReplaceField). Define the order deliberately and test it against sample events before going to production.

Tombstone Messages, Null Payloads, and Delete Event Handling

CDC pipelines generate two Kafka messages for every deleted row.

The first is the delete event: a message with the row’s primary key as the Kafka key and the full delete payload (including the before-image when REPLICA IDENTITY FULL or equivalent is set). This is the CDC notification that the row was removed.

The second is the tombstone: a message with the same Kafka key and a null payload. Its purpose is Kafka log compaction. Compacted topics retain only the most recent message per key. The null-payload tombstone tells compaction that this key’s history can be cleaned up: no “current value” exists for a deleted row.

Consumers that can’t handle null payloads fail on the tombstone. The failure modes vary: deserialization exceptions, NullPointerExceptions in consumer code, or silently dropped messages if the consumer has error handling that swallows them. The result in all cases is that deletes never reach the destination.

Options for handling tombstones:

Filter the tombstones out if the destination doesn’t need them. The Filter SMT can drop any record where value == null. Use this only if you’re certain log compaction isn’t required on the topic, since tombstones are needed for compaction to run correctly.

Route tombstones to a separate topic to process delete signals separately from the main change stream.

Fix the consumer to check for null payload before deserializing. This is the right fix if the consumer is under your control: it makes the consumer correct, rather than dropping information at the connector level.

Test delete handling explicitly. Most CDC integration bugs in production are discovered on the delete path after the initial insert and update path looked fine in testing.

Scaling Connector Tasks and Worker Nodes

Kafka Connect parallelism comes from tasks. One connector can run multiple tasks; each task handles a subset of the source data.

For CDC source connectors, task parallelism maps to table-level parallelism in most implementations. A connector with max.tasks=4 on a 10-table capture might assign 2–3 tables per task. The parallelism benefit is limited by how many tables you’re capturing: you can’t run more tasks than source partitions, and for a single high-throughput table, a single task is common.

"max.tasks": 4

Adding more workers doesn’t automatically increase throughput if tasks are already maxed out. Tasks are the unit of parallelism; workers are the hosts that run them. The practical limit: one task per source partition, distributed across available workers.

For very high-throughput single tables, Kafka Connect’s task model becomes a bottleneck. The connector processes changes from that table in a single task regardless of task count. Sharding the capture (by partition range, or by using a connector with table-level task assignment) is the workaround, and it adds operational complexity.

Error Handling and Dead Letter Queues

Kafka Connect has two failure modes: the connector fails entirely (task stops, you get an alert) or individual records fail silently (task continues, the record is dropped or retried indefinitely).

By default, most Kafka Connect connectors retry failed records. The retry behaviour is configurable:

"errors.retry.timeout": 300000,
"errors.retry.delay.max.ms": 60000

After errors.retry.timeout milliseconds of retrying, the connector either stops the task or routes the failed record to a dead letter queue (DLQ), depending on errors.tolerance:

"errors.tolerance": "all",
"errors.deadletterqueue.topic.name": "cdc-dlq",
"errors.deadletterqueue.context.headers.enable": true

errors.tolerance: all tells the connector to route all failed records to the DLQ rather than stopping the task. errors.deadletterqueue.context.headers.enable writes the exception stack trace and original record metadata into the DLQ message headers, which makes debugging much easier.

For CDC pipelines, the most common sources of individual record failures are schema validation errors (a record from the source has unexpected structure after a DDL change), serialization mismatches (the schema registry rejects a record the connector thinks is valid), and deserialization failures in consumers (the downstream system received a schema it doesn’t recognise).

Monitor the DLQ topic: if records start accumulating there, investigate before they represent a gap in your downstream data. A DLQ that fills silently for a week is worse than a connector that stops immediately.

When Kafka Connect Adds More Complexity Than It Solves

Kafka Connect is a general-purpose connector framework. For CDC specifically, that generality means you’re responsible for:

• Deploying and maintaining a distributed worker cluster
• Managing three coordination topics with the right retention and replication settings
• Defining and ordering SMTs correctly for every connector
• Monitoring offset lag, task failure, and worker health separately from Kafka cluster health
• Handling tombstone messages correctly in every downstream consumer

These problems are tractable in isolation. Each one has a documented solution. The challenge is that you’re solving all of them at once, and the interactions between them are where production incidents come from. An SMT configuration change reorders a transform and a downstream consumer starts seeing unexpected fields. A worker restarts and the offset topic has stale entries. A DLQ fills up and nobody notices until a reporting discrepancy surfaces two weeks later.

The coordination overhead scales with connector count. One connector is manageable. Fifteen connectors, each with their own SMT chain, offset state, and DLQ policy, is a full-time operational concern.

Streamkap’s managed CDC pipeline handles connector deployment, offset management, SMT configuration and task scaling as part of the platform. The connectors catalog covers the available sources and destinations; the Streamkap docs walk through the configuration that stays in your hands versus what the platform manages.

Where to next?

• Compare Streamkap to self-hosted options
• Browse the connector catalog
• Start a free 30-day trial