Apache Iceberg for CDC: Streaming Upserts, Merge-on-Read, and Small-File Management

Streaming CDC into Apache Iceberg is a different problem from batch ETL. Batch loads write large files overnight, you compact occasionally, and query performance stays reasonable. CDC streaming upserts break most of those assumptions. Instead of loading millions of rows in one operation, a CDC pipeline writes hundreds or thousands of small files per hour, each containing a handful of changed rows. The Iceberg table accumulates files faster than most compaction setups can clear them, and query performance degrades quietly until someone notices that a scan that used to take 30 seconds now takes 5 minutes.

The write-time decision (merge-on-read or copy-on-write) determines how badly this degradation occurs and how much compaction work is needed to stay ahead of it.

Why Iceberg Needs a Different CDC Write Strategy

Iceberg organizes data into immutable files. Every write (insert, update, delete) produces one or more new files. That’s the design: immutability is what enables time travel, snapshot isolation and concurrent reads.

The problem for streaming CDC is file count. A batch load writing 10 million rows might produce 10–20 well-sized Parquet files. A CDC pipeline writing 10 million individual change events over the course of a day (not unusual for an active OLTP table) might produce 10,000 files, one per commit or more. Iceberg needs to track every one of those files in its table metadata, and every query needs to consider all of them when computing the scan plan. The query overhead is proportional to file count, not just data volume.

Two write strategies handle this differently.

Copy-on-write (CoW) processes each change by rewriting every data file that contains the affected row. An UPDATE rewrites the original data file with the new row value in place. A DELETE rewrites the data file with the row removed. The result is a small number of large, clean data files after each operation.

Merge-on-read (MoR) doesn’t touch the existing data files. Instead, it writes a separate delete file that records which rows were changed or removed, and a new data file with the updated values. Queries then merge the data files and delete files at read time to produce the current view.

Neither is universally better. CoW keeps data files clean at the cost of amplified writes (rewriting an entire file to change one row). MoR keeps writes cheap at the cost of growing delete file counts and read-time merge overhead.

Merge-on-Read: How CDC Deltas Land in Iceberg

For streaming CDC, MoR is almost always the right starting point. CDC writes are by definition high-frequency and low-volume per event: rewriting large data files for each individual update (CoW) would generate enormous write amplification and file churn.

With MoR, an UPDATE from the CDC stream produces two files:

1. A data file with the new row value
2. An equality-delete file identifying the row to be replaced by primary key

At query time, Iceberg’s query engine reads both the existing data files and the delete files, applies the deletes, and returns the merged result. The query plan includes every delete file in scope.

Here’s what degrades as delete files accumulate. Each snapshot can reference many delete files. A table with one week of active CDC writes and no compaction might have thousands of delete files, each containing a handful of entries. The query engine must open and process all of them to compute the current table state. The I/O cost grows with delete file count, not just with total data volume.

A key diagnostic: check the delete file count in iceberg_snapshots or via the Iceberg metadata tables:

SELECT snapshot_id, added_delete_files_count, total_delete_files_count
FROM iceberg_snapshots
ORDER BY committed_at DESC
LIMIT 10;

When total_delete_files_count grows faster than compaction can reduce it, queries slow down. The threshold varies by storage system, but tables with more than 1,000 active delete files typically show noticeable scan overhead.

Copy-on-Write: When It Makes Sense for CDC

CoW has one clear advantage over MoR: after each write, the data files are clean. Queries see no delete files to merge; the scan is simply the data files in the current snapshot. Read performance is predictable regardless of how many updates have happened.

The trade-off is write amplification. For a CoW table, changing one row in a 256MB Parquet file means rewriting the entire 256MB file to produce a new snapshot with the updated value. For a table with 100 files averaging 256MB each, a batch of updates touching every file produces 100 new files, totalling 25.6GB written to change perhaps a few MB of actual row data.

For CDC workloads, CoW is worth considering in two scenarios:

Low-churn reference tables. Tables that receive a steady trickle of updates (configuration data, lookup tables) with far more reads than writes tolerate CoW well. The write amplification is modest because the change volume is small; queries benefit from clean files with no delete overhead.

Large batch updates with infrequent streaming. Some CDC patterns batch changes (e.g., end-of-business-day settlement records) rather than streaming continuously. If updates arrive in large batches rather than as individual events, CoW’s amplification is less extreme per operation (fewer rewrites, larger batches) and the clean file state between batches makes read performance predictable.

For a mixed workload (some high-update tables, some low-update), you can set the write mode per table rather than globally.

Compaction Strategy for Streaming CDC Workloads

Regardless of write mode, streaming CDC tables need compaction. The question is how to schedule it so it keeps file counts manageable without blocking concurrent writes.

The two Iceberg compaction strategies are bin-pack and sort rewrite.

Bin-pack rewrite groups small files into target-size files (typically 128MB–512MB). For CDC tables generating many small files, bin-pack compaction is the right tool: it reduces file count directly, with no sorting overhead. The relevant settings:

CALL system.rewrite_data_files(
  table => 'db.orders',
  strategy => 'binpack',
  options => map(
    'target-file-size-bytes', '134217728',   -- 128MB
    'min-input-files', '20'
  )
);

min-input-files prevents compaction from running on partitions that don’t have enough small files to consolidate. Setting it too low wastes compaction resources on partitions that are already healthy; too high lets small-file accumulation run unchecked.

Sort rewrite re-sorts data by a specified column order as it compacts, which improves query performance for range scans on the sort key. For CDC tables where queries filter by event timestamp or primary key, sort compaction trades higher write cost for lower query cost. Run sort compaction less frequently than bin-pack (weekly, or as a catch-up after a major backfill), not as the routine cleanup.

For concurrent writes, use the partial-progress.enabled option to allow compaction to commit in segments rather than as a single transaction. This reduces conflict with ongoing CDC writes:

CALL system.rewrite_data_files(
  table => 'db.orders',
  strategy => 'binpack',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10',
    'min-input-files', '20'
  )
);

For compaction failures and recovery after a degraded table state, the lake format compaction and recovery guide covers the full operational runbook.

Snapshot Expiry and Orphan File Cleanup

Compaction rewrites data files, but it doesn’t remove the old files or the snapshots that reference them. Each compaction operation creates a new Iceberg snapshot; the old snapshot still references the original small files. Until you expire old snapshots, those files remain on storage even though no current snapshot needs them.

Expire snapshots on a schedule that keeps enough history for your time-travel requirements. For a CDC table where you need 7 days of point-in-time query support:

CALL system.expire_snapshots(
  table => 'db.orders',
  older_than => TIMESTAMPADD(DAY, -7, NOW()),
  retain_last => 5
);

retain_last ensures at least N snapshots are kept regardless of age. This prevents accidentally expiring the entire table history if the pipeline is quiet over a weekend and the scheduled job runs aggressively.

After snapshot expiry, orphan file cleanup removes any files on storage that no snapshot references. These accumulate from partial writes (writes that failed after creating files but before committing the snapshot), from compaction operations that produced files but were interrupted, and from manual file manipulations.

CALL system.remove_orphan_files(
  table => 'db.orders',
  older_than => TIMESTAMPADD(HOUR, -48, NOW())
);

The older_than threshold here should be conservative: set it to at least 24–48 hours to avoid deleting files from in-progress writes. A file that’s part of an active write operation (started but not yet committed) will look like an orphan until the commit completes. Deleting it mid-write corrupts the pending snapshot.

For streaming CDC tables, run these maintenance operations as scheduled queries against your query engine (Spark, Trino, Flink SQL, or whatever runs your Iceberg catalog). The sequence is always: compact first, then expire snapshots, then remove orphans. Reversing the order is safe but wastes I/O on files that compaction will just rewrite anyway.

Treating these three operations as a single maintenance pipeline, scheduled together, keeps CDC tables healthy without requiring manual intervention as write volume grows.

Connecting Streamkap to Iceberg

Streamkap’s Iceberg connector writes CDC events using MoR by default, with equality deletes for updates and deletes. The connector supports two ingestion modes:

Append mode writes all CDC events as new rows without attempting to reconcile deletes at write time. This is the lowest-latency path: events hit Iceberg as soon as they arrive. A downstream view or compaction job handles the current-state computation.

Upsert mode applies updates and deletes using Iceberg’s merge semantics, maintaining a current-state table without requiring a separate deduplication view. Upsert mode has higher write overhead because it needs to track which rows to delete.

The two modes are incompatible at the table level: you can’t switch between them on a live table without creating a new connector targeting a new destination table. Choose based on whether your consumers need a current-state table or are equipped to handle the event-log pattern.

The Streamkap Apache Iceberg documentation covers both modes, Glue catalog setup, and the compaction recommendations for each volume tier.

For a broader introduction to Iceberg’s architecture and how it fits into a real-time data stack, the Apache Iceberg guide covers the table format fundamentals.

Where to next?

• Iceberg connector overview
• Streamkap Apache Iceberg documentation
• Apache Iceberg guide
• Start a free 30-day trial