DynamoDB Streams to Kafka: Stream View Types, Shard Iteration, and Production CDC Setup

DynamoDB CDC fails in a specific way that surprises teams coming from relational database CDC. With Postgres or MySQL, the main risk is a WAL segment or binlog purge: the connector falls behind, the log file gets recycled, and you need a re-snapshot. DynamoDB has the same cliff, but it’s fixed at 24 hours regardless of table size, write volume, or any configuration you control.

The other common failure happens at setup, not during operation. A team enables DynamoDB Streams and connects a CDC connector, but picks the wrong stream view type. The connector starts receiving events, the metrics look fine, but the events are missing the before-state for updates and deletes, which means downstream tables quietly get incorrect data.

Both failures are preventable. They require understanding how DynamoDB Streams works at a lower level than most tutorials cover.

How DynamoDB Streams Differ from Log-Based CDC

Postgres, MySQL, and SQL Server CDC all read from a sequential write-ahead log. DynamoDB Streams doesn’t have a WAL. Instead, it’s a time-ordered sequence of item-level change events, partitioned into shards that track the table’s own partition structure.

Each DynamoDB table partition maps to one or more stream shards. When DynamoDB splits a table partition (because write volume on that partition exceeds its allocated throughput), the corresponding stream shards also split. Each shard is an independent ordered sequence: events within a shard are ordered, but ordering across shards is not guaranteed.

This shard structure has two practical consequences for CDC connectors.

First, the connector must track its position per shard, not as a single global sequence number. A connector that checkpoints only a global offset will lose events when shards split, because the new shards start at the split point, not at the beginning.

Second, shards close after they’ve been inactive for about four hours (when a partition is no longer actively written to). A connector needs to handle closed shards, open their successors, and continue without gaps.

The fixed 24-hour retention applies at the shard level: each stream record expires 24 hours after it’s created, regardless of whether any consumer has read it.

Stream View Types and Why the Choice Matters

DynamoDB Streams offers four stream view types, set when you enable streams on a table:

KEYS_ONLY: each event contains only the primary key attributes (partition key and sort key, if any). No attribute values, no before or after state. This is the lowest overhead option but is insufficient for CDC: a connector can’t tell what changed or what the item contained.

NEW_IMAGE: each event contains the item’s state after the change. Inserts and updates carry the full item. Deletes carry only the key (because the item no longer exists). A CDC connector using NEW_IMAGE can apply inserts and updates, but for deletes it only knows which key was deleted, not what the item contained before deletion.

OLD_IMAGE is the inverse. Each event carries the item’s state before the change, so deletes include the full pre-deletion item but inserts carry only the key. Rarely useful for CDC on its own.

NEW_AND_OLD_IMAGES: each event contains both the before and after state of the item. Inserts include the new item with a null old image; updates include both states; deletes include the old item with a null new image. This is the view type CDC connectors need.

If you connect a CDC connector to a stream with KEYS_ONLY or NEW_IMAGE, the connector will process events without error. The events themselves are structurally valid. The problem shows up downstream: deletes arrive without the item data, updates may land as inserts if the destination can’t match them to existing records by key alone, and any downstream system tracking the before-image of changes will silently receive incomplete data.

Change the view type on an existing stream via the AWS CLI or console. You must disable the stream, re-enable it with the new view type, and then reconnect the connector. Disabling and re-enabling resets the stream, so existing records in the old stream are lost.

Shard Iterators and the 24-Hour Retention Constraint

To read from a DynamoDB Stream shard, a connector requests a shard iterator. There are four iterator types:

• TRIM_HORIZON: start from the oldest available record in the shard.
• LATEST: start from the most recent record, discarding all history.
• AT_SEQUENCE_NUMBER: start at a specific sequence number (used on restart when the connector has a stored checkpoint).
• AFTER_SEQUENCE_NUMBER: start after a specific sequence number.

On initial setup, a connector uses TRIM_HORIZON to read all available history and then switches to streaming live events. On restart, it uses AT_SEQUENCE_NUMBER with its stored checkpoint to resume.

The 24-hour cliff hits when a connector is offline longer than the stream’s retention window. The sequence number stored in the checkpoint still exists as metadata, but the records it references have expired. Calling GetRecords on an AT_SEQUENCE_NUMBER iterator for an expired position returns a trimmed error: the shard has been trimmed past that point.

When this happens, the connector has two options. It can start from TRIM_HORIZON on each shard, which gives it everything still within the 24-hour window but creates a gap for the period between the last checkpoint and the start of the available window. Or it can do a fresh table snapshot to bring the destination back to a consistent state and then stream from LATEST.

The correct choice depends on whether the destination can tolerate a gap. Most CDC pipelines can’t: a gap means some changes were missed, and downstream data is inconsistent. For most teams the right answer is re-snapshot.

The operational response to this risk is straightforward: keep your connector’s checkpoint lag well inside 24 hours. Alert on GetRecords.IteratorAgeMilliseconds before it reaches 12 hours. That gives you time to investigate and restart the connector before any records expire.

IAM Permissions and Global Tables Considerations

DynamoDB Streams uses IAM for access control rather than database-level credentials. The connector needs a set of permissions on the table’s stream:

{
  "Effect": "Allow",
  "Action": [
    "dynamodb:DescribeStream",
    "dynamodb:GetRecords",
    "dynamodb:GetShardIterator",
    "dynamodb:ListStreams",
    "dynamodb:DescribeTable"
  ],
  "Resource": [
    "arn:aws:dynamodb:us-east-1:123456789:table/orders",
    "arn:aws:dynamodb:us-east-1:123456789:table/orders/stream/*"
  ]
}

Scope the resource to the specific table and stream ARN. Wildcard resources (arn:aws:dynamodb:*:*:table/*) are a common shortcut during setup that should be narrowed before production.

Global Tables add a wrinkle. A DynamoDB Global Table replicates data across multiple AWS regions, with each region accepting writes. When a write originates in eu-west-1 and replicates to us-east-1, the replicated event appears in the us-east-1 stream as if it were written there. A CDC connector reading from us-east-1 will see both locally-originated writes and writes that originated in other regions.

This creates a potential duplicate-delivery problem if you’re running connectors in multiple regions. A write that originated in eu-west-1 will appear in the eu-west-1 stream and, after replication, also appear in the us-east-1 stream. If both streams feed the same Kafka topic, the change event arrives twice.

The DynamoDB Streams event includes a userIdentity field that identifies the originating region for replicated writes. A connector aware of Global Tables can filter on this field to process only locally-originated events, relying on the connector in each region to handle its own writes. Not all connectors implement this filtering; check the documentation before deploying a multi-region setup.

Production Monitoring: IteratorAgeMilliseconds and Shard Count

The key CloudWatch metric for a DynamoDB Streams CDC pipeline is GetRecords.IteratorAgeMilliseconds. This measures the age of the newest record returned by a GetRecords call relative to the current time. When the connector is keeping up, this number is small, typically under a second for a low-latency pipeline. When the connector falls behind, it grows.

Set two alert thresholds:

• Warning at 4 hours: the connector is falling behind. Investigate before the gap grows further.
• Critical at 12 hours: the connector is at risk of losing records before it can process them. Requires immediate action.

Neither threshold is a hard limit from AWS; they’re lead indicators giving you time to respond before the 24-hour retention cliff is reached.

Two other metrics to watch:

GetRecords.SuccessfulRequestLatency: high latency here indicates the connector is spending more time waiting for GetRecords responses, which can slow it down. DynamoDB Streams has throughput limits per shard (2 MB/second read rate, 5 read transactions per second per shard), and a connector hitting those limits will back off and slow its consumption rate.

Shard count: DynamoDB adds new shards when it splits partitions during table scaling. A connector that isn’t discovering new shards will miss all events written to those partitions. Monitor for shard splits (a CloudWatch event fires when a split occurs) and confirm your connector is consuming all active shards after a scaling event.

For tables with predictable traffic patterns, table auto-scaling triggers shard splits predictably during peak periods. A CDC connector that doesn’t handle shard splits gracefully will have regular data gaps around traffic peaks.

Connecting Streamkap to DynamoDB

Streamkap’s DynamoDB connector validates that the stream view type is set to NEW_AND_OLD_IMAGES on connection, surfaces an error if it isn’t rather than connecting to an unsuitable stream, and handles shard discovery automatically when the table scales. On restart, it checks the age of its stored sequence number checkpoints against the current stream before attempting to resume, so you get a clear signal about whether recovery requires re-snapshotting rather than discovering a gap mid-stream.

Streamkap supports DynamoDB tables in any AWS region, DynamoDB Global Tables with per-region filtering, and connects via IAM role assumption for credential management. The Streamkap DynamoDB documentation covers IAM setup, the initial snapshot process for large tables, and the monitoring configuration for IteratorAgeMilliseconds alerts.

Where to next?

• DynamoDB connector overview
• Streamkap DynamoDB documentation
• Start a free 30-day trial