Postgres CDC to Kafka: End-to-End Pipeline Setup and Configuration Guide

Setting up a Postgres CDC to Kafka pipeline looks straightforward until it isn’t. It works fine in staging, then the connector restarts after a deployment, the replication slot had been holding WAL all weekend, and Monday morning the primary is out of disk.

This guide walks through the setup from the beginning: how PostgreSQL’s logical replication mechanism feeds a Kafka topic, which connector settings actually matter, how schema changes flow downstream, and what you need to watch before that disk-full scenario becomes your Monday morning.

How PostgreSQL Logical Replication Feeds a Kafka Topic

PostgreSQL writes every committed change to its Write-Ahead Log before any data files are updated. Logical replication builds on that foundation by decoding WAL entries into structured row-level events: inserts, updates and deletes, each carrying the table name, operation type and affected row data.

A replication slot is the bookkeeping mechanism. It tracks exactly how far a connected consumer has read, and it prevents PostgreSQL from reclaiming WAL segments until the consumer has acknowledged them. That’s what makes CDC resumable after restarts. It’s also what causes WAL to pile up when a consumer goes offline, since the slot keeps holding those segments until something reads and confirms them.

Between the slot and your connector sits a logical decoding plugin. Two options are common:

pgoutput ships with PostgreSQL 10+ and needs no extension install. It uses PostgreSQL’s publication model, where you define explicitly which tables to capture. For any new pipeline, pgoutput is the right choice.

wal2json and decoderbufs are extension-based alternatives that predate pgoutput. You’ll encounter them in older setups, but pgoutput has been the standard since PostgreSQL 10 and there’s no reason to start with the extensions today.

From the connector’s perspective: it connects via the PostgreSQL replication protocol, opens the logical replication slot, and streams decoded change events. Each event carries enough context to reconstruct the full row state at the destination.

Connector Configuration: Slot, Plugin and Publication

Four settings shape how the connector behaves. Getting any of them wrong costs you either correctness or operational pain.

slot.name is the identifier for the PostgreSQL replication slot. Use something stable and descriptive, like streamkap_pgoutput_slot. A connector that restarts and reconnects to the same slot name picks up exactly where it left off. A connector configured with a new slot name on each restart starts a fresh snapshot every time, which is expensive for tables with millions of rows.

plugin.name should be pgoutput on any PostgreSQL 10+ instance. If you’re on an older self-hosted setup using an extension-based decoder, plan the migration to pgoutput; it’s significantly more maintainable.

publication.name references a PostgreSQL publication. Create it before starting the connector:

-- Capture specific tables
CREATE PUBLICATION streamkap_pub FOR TABLE orders, customers, products;

-- Or capture everything
CREATE PUBLICATION streamkap_pub FOR ALL TABLES;

The publication controls which tables the slot captures. Creating it explicitly (rather than letting the connector create it automatically) means you control the scope. One thing to know: a FOR ALL TABLES publication can’t later be narrowed to specific tables. If you need to exclude tables later, you’ll drop the publication and create a new one, which means a re-snapshot of all included tables.

snapshot.mode determines what happens at connector startup. initial takes a full table snapshot and then switches to streaming. initial_only takes the snapshot and stops. never skips the snapshot and starts from the current WAL position. For a new pipeline connecting to an existing database, initial is almost always what you need.

REPLICA IDENTITY and Full Row Images

By default, PostgreSQL’s UPDATE and DELETE events in the logical stream carry only the primary key of the changed row. That’s usually enough if your destination is doing keyed upserts. But it means you don’t get the complete before-image of a row.

REPLICA IDENTITY FULL changes that:

ALTER TABLE orders REPLICA IDENTITY FULL;

With FULL set, UPDATE events carry both the complete before and after row, and DELETE events include the full row rather than just the key. This matters for audit trails, for destinations that don’t support keyed deletes, and for any downstream consumer that needs the before-image to compute deltas.

The cost is WAL volume. FULL identity writes the entire row on every change, which can meaningfully increase WAL generation on tables with wide rows or high update rates. Enable it selectively on tables where the before-image is genuinely useful, not as a default for all tables.

Schema Evolution Through the Replication Stream

PostgreSQL doesn’t send explicit schema-change events through the logical replication stream. When you add a column with ALTER TABLE, the connector detects the change when it processes the first event from that table that includes the new column.

How this reaches Kafka depends on whether you’re using a schema registry:

With a schema registry, the connector registers a new schema version when it sees the new column set. Consumers configured for schema evolution can handle nullable column additions as backward-compatible additions. Type changes and non-nullable column additions are breaking and require consumer coordination before the ALTER TABLE reaches production.

Without a schema registry, the connector typically embeds a schema envelope in each message. Consumers parse the schema per-message. It’s less efficient but avoids registry coupling.

The practical rule: add new columns as nullable. Test consumer behaviour against the new schema version before any breaking change (type changes, column removals) reaches a production database.

Slot Lag, WAL Accumulation and Production Health

Two lag metrics matter for this pipeline, and they’re related.

Replication slot lag measures the gap between PostgreSQL’s current WAL write position and the last position the connector confirmed. Query it directly:

SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots;

When lag grows, PostgreSQL retains WAL segments on disk for the slot. The risk is disk exhaustion on the primary. Set an alert when any slot’s retained WAL crosses 50% of your max_slot_wal_keep_size (PostgreSQL 13+), and a critical alert at 80%.

Kafka consumer lag measures how far behind the downstream consumer is on the Kafka topic partitions. High consumer lag feeds back to slot lag: a slow consumer slows the connector’s acknowledgments, which prevents the slot from advancing.

The two lags are related but distinct. Slot lag grows when the connector is offline or can’t write to Kafka. Consumer lag grows when the connector is writing but the downstream consumer can’t keep up. Both need monitoring, but the remediation is different.

For a more thorough treatment of slot lifecycle management, WAL retention tuning and HA failover behaviour, PostgreSQL WAL Slot Management: Prevention and Recovery at Production Scale covers those in detail.

Heartbeat Tables and Idle WAL Advancement

Replication slots have a subtle problem on databases with low write volumes: if no tables change, no new WAL is written, and the slot’s confirmed LSN doesn’t advance. The slot stays open, but the connector has nothing new to confirm. On very quiet databases, this can leave the slot sitting idle for extended periods.

The risk surfaces when the connector restarts after an outage on a quiet database. The slot has accumulated some WAL (background processes write occasional system records), but there’s no way to tell how stale the connector’s view is just from the slot lag metric.

Heartbeat tables solve this by giving the connector something to confirm on a regular schedule. A heartbeat record — a simple counter or timestamp updated every minute — forces WAL events that the connector can acknowledge, keeping the confirmed LSN current even when application tables are idle.

Most CDC connectors support heartbeat configuration directly:

"heartbeat.interval.ms": 10000,
"heartbeat.action.query": "INSERT INTO streamkap_heartbeat (ts) VALUES (NOW()) ON CONFLICT (id) DO UPDATE SET ts = NOW()"

The heartbeat table needs to exist in the captured database and be included in the publication:

CREATE TABLE public.streamkap_heartbeat (
  id    INTEGER PRIMARY KEY DEFAULT 1,
  ts    TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
ALTER TABLE public.streamkap_heartbeat REPLICA IDENTITY FULL;

The connector updates this table on its configured interval. The resulting WAL events flow through the slot, the connector reads and acknowledges them, and the confirmed LSN stays current. On reconnect, the connector can start from a recent offset rather than potentially re-reading a large backlog of system WAL.

Beyond slot health, heartbeat events also give you a latency signal. If heartbeats stop arriving at the Kafka topic, the connector has stalled. Alerting on heartbeat topic lag gives earlier warning than waiting for application table events to surface the problem.

One operational note: the heartbeat table itself shows up as CDC events in the Kafka topic. Downstream consumers should filter these out unless they’re explicitly monitoring them. Most connectors write heartbeat events to a dedicated topic (separate from application table topics) by default.

Routing Multiple Tables to Multiple Topics

By default, a CDC connector publishes each table to a topic named <server>.<schema>.<table>. That works for simple cases. Production pipelines usually need more control.

Per-table topic routing assigns each table to a specific topic name via a single-message transform (SMT). You configure the routing at the connector level, before events reach Kafka. This keeps topic naming consistent regardless of where the connector is running or how it was restarted.

Partition keying controls which partition an event lands on within a topic. The default is the primary key, which keeps all changes for a given row on the same partition. That ordering guarantee is what makes downstream upserts correct: if two updates to the same row arrive, they’re processed in the order they were written to PostgreSQL.

For high-throughput tables, think about partition count before you create the topic. More partitions allow more parallel consumers, but adding partitions after the fact on a live Kafka topic requires a careful migration. The Streamkap performance tuning guide covers the safe partition increase procedure.

Streamkap’s PostgreSQL connector handles slot creation, heartbeat configuration, monitoring and reconnection automatically. You define the tables and destination topics; the connector manages the replication slot lifecycle and WAL advancement without manual intervention.

Where to next?

• Set up your PostgreSQL source in Streamkap
• PostgreSQL connector overview
• PostgreSQL WAL slot management in production
• Start a free 30-day trial