MongoDB CDC: Change Streams, Replica Set Requirements, and oplog Sizing for Production

Most CDC connectors track their position against a write-ahead log. Postgres has replication slots, MySQL has the binlog, SQL Server has the LSN sequence. MongoDB uses something structurally similar but materially different: the oplog, a capped collection that lives inside every replica set member.

Understanding that distinction matters before you connect a CDC connector. Oplog rollover, resume token expiry, and incorrect pipeline configuration all stem from treating MongoDB change streams as if they worked like a WAL.

How MongoDB Change Streams Differ from WAL-Based CDC

MongoDB change streams are built on the oplog, a special collection in the local database that records every write operation committed to the cluster. Unlike a WAL, the oplog is a capped collection with a fixed maximum size. When it fills, the oldest entries are overwritten.

A CDC connector opens a change stream and receives an event for each document write: inserts, updates, replaces, and deletes. The connector’s position in that stream is a resume token, a BSON object that encodes the oplog entry’s timestamp and transaction identifier.

Two consequences follow from this architecture.

First, change streams require a replica set. A standalone MongoDB instance has no oplog. If you’re running MongoDB without --replSet, you cannot use change streams. This trips up development environments where a single mongod with no replication is common.

Second, the resume token is only valid while the corresponding oplog entry still exists. If the oplog wraps around past a token, the token becomes invalid. This happens when writes exceed the oplog’s capacity faster than the connector can consume them, or when the connector is offline long enough for the oplog to cycle past its last position. MongoDB returns a ChangeStreamHistoryLost error, and the only way forward is a full snapshot of the affected collections.

Replica Set Prerequisites and oplog Configuration

The minimum requirement for change stream CDC is a replica set with at least one member. A three-member set (one primary, two secondaries) is the production standard because it gives you an automatic primary failover path. A single-member replica set works for CDC but has no failover.

Check whether your instance is running as a replica set:

rs.status()

If you get MongoServerError: not running with --replSet, you’re on a standalone. Converting a standalone to a replica set requires a restart with --replSet set in mongod.conf.

The oplog size is the more operationally critical setting. By default MongoDB allocates a slice of available disk space to the oplog, typically 5% of free disk or a minimum of 1 GB, whichever is larger. On a database with a high write rate, 1 GB of oplog can represent less than an hour of history.

Check the current oplog window:

rs.printReplicationInfo()

The output shows the oplog size in megabytes and an estimated window in hours based on recent write rate. The “estimated” part is important: write rates are rarely constant, and a log ingestion spike can shrink the window significantly.

To change the oplog size on MongoDB 4.4+, no downtime is needed:

db.adminCommand({ replSetResizeOplog: 1, size: 51200 }) // 50 GB in MB

On older versions, changing oplog size requires taking each member offline in sequence and restarting it with the new --oplogSize value. For a busy production cluster, plan this change carefully.

The right oplog size depends on your write rate and your worst-case connector downtime tolerance. If peak write rate generates 5 GB of oplog per day, and you want to survive a 72-hour connector outage (a long weekend plus a deployment failure), you need 15 GB or more. Most teams size it to cover at least twice their planned maintenance windows.

Change Stream Filtering and Pipeline Configuration

By default, a change stream on a collection returns every event: inserts, updates, replaces, deletes, and schema-level invalidate events. For CDC purposes you usually want all of these, but the change stream API accepts an aggregation pipeline for filtering.

Filter by operation type to capture only inserts and updates:

[{ $match: { operationType: { $in: ["insert", "update", "replace"] } } }]

One option that matters for CDC is fullDocument. By default, an update event carries only the fields that changed (a delta), not the complete document. For most CDC destinations this is insufficient: you need the full post-image to apply the change correctly.

Enable updateLookup to receive the full document on update events:

collection.watch([], { fullDocument: "updateLookup" })

With updateLookup, MongoDB performs an additional read to fetch the current document state and includes it in the change event. This trades some latency and source-side read load for complete events downstream. One edge case: if the document was deleted after the update but before the lookup completes, the fullDocument field in the event will be null.

MongoDB 6.0 added fullDocumentBeforeChange, which returns the pre-image of the document. This requires enabling changeStreamPreAndPostImages on the collection:

db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: { enabled: true } })

Not all connectors use this option, but when your pipeline requires explicit before and after states for every update, it’s the cleanest solution.

Resume Tokens and Restart Semantics

The resume token is what allows a connector to pick up where it left off after a restart. Every change event carries a _id field containing the token: a BSON document that encodes the oplog timestamp and transaction ID.

A connector stores its last-processed resume token, usually in Kafka offsets or a local state store. On restart, it opens a new change stream with resumeAfter: <stored_token> and MongoDB seeks to the matching oplog position.

Two conditions break this:

ChangeStreamHistoryLost: the oplog entry referenced by the resume token no longer exists. The oplog has wrapped past that position. At this point, resumeAfter fails and the connector must re-snapshot the collections it was watching and start a new change stream from the current position.

Invalidate events: a change stream closes automatically when the watched collection is dropped or renamed, or when the database is dropped. The last event in the stream is an invalidate event. The connector needs to handle this case explicitly. Some connectors re-open a new stream; others surface it as an error requiring operator action.

To distinguish between the two scenarios in monitoring, check the error type. ChangeStreamHistoryLost is recoverable by re-snapshotting. An invalidate event has a specific operationType: "invalidate" and warrants investigation into why the collection disappeared.

One practical rule: if your connector restarts after a long outage, check the oplog window before attempting to resume. If the outage duration was close to or exceeded the current oplog window, assume the token is gone and trigger a re-snapshot rather than waiting for the ChangeStreamHistoryLost error.

Production oplog Monitoring and Retention Alerts

The oplog window is your primary operational safety margin for change stream CDC. Monitor it actively.

Two queries to run from any replica set member:

// Current oplog window estimate
rs.printReplicationInfo()

// Member-level replication status
rs.printSecondaryReplicationInfo()

The second command shows replication lag per secondary. A secondary that’s significantly behind the primary is consuming more of the oplog, holding events it hasn’t replicated yet and preventing them from being safely purged. If a secondary goes offline, it holds the oplog in place on the primary, which can cause disk pressure if the cluster is write-heavy.

Key CloudWatch or Atlas metrics to alert on:

oplog window hours: set an alert when this drops below two times your longest planned maintenance window. This gives you time to respond before a connector outage would cause a gap.

replication lag (secondary lag vs primary): alert if any secondary is more than a few minutes behind. Extreme lag indicates a secondary is struggling to keep up with write volume, and the oplog is providing a buffer that could expire.

connector task restarts: a spike in task restarts without a corresponding deployment is the first sign that the connector is having trouble. Investigate before the underlying gap causes a re-snapshot.

For Atlas clusters, set the minimum oplog window through the Cluster Settings UI. This overrides the automatic sizing and guarantees MongoDB maintains at least that many hours of history, regardless of write rate and disk usage.

Connecting Streamkap to MongoDB

Streamkap’s MongoDB connector handles replica set detection, fullDocument configuration, resume token storage, and restart recovery. On connection, it validates that the target is a replica set member rather than a standalone, surfaces a clear error if change streams aren’t available, and sets fullDocument: "updateLookup" by default so update events carry complete documents.

During snapshot, the connector uses a consistent read against the collection to capture existing documents. For large collections, the snapshot runs incrementally so the initial burst of events doesn’t arrive all at once.

On restart, the connector checks its stored resume token against the current oplog state before attempting to resume. If the token is close to the edge of the available window, it surfaces a warning rather than proceeding silently and potentially hitting ChangeStreamHistoryLost mid-stream.

Streamkap supports self-hosted MongoDB, MongoDB Atlas, and AWS DocumentDB (with change streams enabled). The Streamkap MongoDB documentation covers required user roles, Atlas-specific connection configuration, and the re-snapshot workflow when resume tokens expire.

Where to next?

• MongoDB connector overview
• Streamkap MongoDB documentation
• Start a free 30-day trial