Your CDC pipeline saved its place. Why is the history gone?

When a CDC pipeline stops, it is understandable to expect that restarting it will allow it to catch up. For that to work, the history it needs must still be available.

Let’s say that history is gone. Instead of processing a backlog of changes, recovery may require reading the source tables again and rebuilding the destination. Even if the interruption is brief, the amount of data to copy may be large.

Let’s follow what happens to the pipeline’s saved position.

The bookmark survives. Does the history?

A connector records how far it has read through the source’s change history so it knows where to resume. Think of it as a bookmark. This bookmark may contain several values that together identify where reading should continue.

While the connector is stopped, the source may still be changing. When the connector reconnects, it uses the bookmark to try to resume reading where it left off.

Just because you have a saved bookmark doesn’t mean the history is still there when you return.

How long the source retains history depends on the database and its configuration. A source may remove old changes after a time limit or to stay within a storage limit. It may also retain history for a stalled consumer until storage runs out. The source needs to both protect itself from unlimited growth and preserve enough history so the connector can recover from a stall.

If the connector was already behind when it stopped, the history it needs is older than the outage itself. A short interruption can be enough to lose access to required history when the pipeline was already close to its retention boundary.

In my analysis of 1,965 public CDC issues, 27 percent matched the offset and position category, the largest of the eight categories I tracked. Offsets and positions are the technical terms for the bookmarks described here. That broad category includes more than expired history. It does not tell us how often an outage requires a rebuild, but it gives us a reason to examine what makes a saved bookmark usable.

When restarting becomes rebuilding

An unusable bookmark does not automatically mean all the data must be copied again. The history that is required may survive elsewhere. Missing connector state may be repairable. A failover may preserve enough state to resume, depending on the system and its configuration.

If the history needed to resume from the bookmark is gone and cannot be recovered, reconnecting to the source won’t bring it back. Processing new changes alone won’t fix the gap in the destination.

Restoring the destination may require copying the affected source tables again in a new snapshot. Recovery must also account for ongoing changes and remove stale rows.

Recovery now involves reading whole tables, including rows that never changed during the outage. A short interruption can mean a large amount of work.

The connector is back, but the copy is still running

Reading those tables consumes I/O, CPU, and network capacity that production traffic may also need. If the source continues changing during recovery, the pipeline must also account for those changes. Meanwhile, the destination has to accept the copied rows.

A rebuild’s duration depends on the dataset, available throughput, concurrent writes, and implementation. It is not a fixed multiple of the time the outage took. A short interruption can lead to a long recovery because the interruption triggered a different amount of work.

An interruption during the snapshot raises another question: how much completed work survives? Some snapshot modes restart after failure. Others save progress and resume by table or chunk. Saving a cursor preserves progress, but the snapshot process must be able to handle changes made while the copy is underway.

There is another problem even when every surviving row has been copied successfully.

Copying the rows does not remove the missing ones

What if a row had already been copied to the destination, but was deleted from the source while the connector was stopped? By the time the connector returns, the history containing that delete is gone. If recovery only inserts or updates rows in the existing destination, nothing in that operation removes the old row.

This means the copy can finish and still leave the destination wrong.

The recovery process needs a way to reconcile those absences. It can replace the affected destination data, or it can identify and remove stale rows after a successful refresh. One approach marks the rows with a snapshot generation and sweeps older rows after that snapshot completes. That sweep must be coordinated with ongoing changes so it does not delete valid new rows, and a partial snapshot must not be treated as a complete one.

Applying the copied rows also needs to be safe when work is repeated. Keyed upserts can prevent duplicate rows, but deletes and event ordering still matter. Replaying an older value after a newer one must not leave the destination with stale data.

A successful rebuild therefore means more than finishing the scan. It means bringing the destination into agreement with the source.

What even a successful rebuild cannot restore

Now consider a different row. This one was created and deleted entirely during the missing interval. It was never captured, and it no longer exists when the new snapshot runs.

A later snapshot cannot tell you that row existed. It also cannot reconstruct intermediate values of a row updated several times during the gap.

With appropriate reconciliation and capture of ongoing changes, a snapshot can restore the current state. It cannot restore event history that is no longer available anywhere.

For a current-state mirror, reconciliation may meet the recovery requirement. For an audit history or a consumer that acts on individual events, the missing events will remain lost even after the tables agree.

Establish the recovery path before the outage

When the pipeline comes back, can it catch up from its saved position, or does it need to rebuild the destination?

The answer requires you to separate capture from delivery. A connector may already have written changes to a durable log while the destination was unavailable. If that log retains everything the destination needs and the destination has a usable checkpoint, delivery can resume without rereading the source.

Rebuilding a lost destination from that log is also possible if it contains an initial copy of the data and all the changes needed to bring it up to date. That removes the need to copy from the source, but it may still require processing the entire retained history again. Also, a downstream log cannot supply changes that capture never recorded.

Before relying on a recovery procedure, establish three things:

1. Where does it read from? Which retained history can it use, and under what conditions does recovery require another source snapshot?

2. What completed work survives? If replay or snapshotting is interrupted, where does it resume, and how does it handle repeated writes?

3. What does completion guarantee? Can recovery restore what the application needs, whether that is current state, complete event history, or both?

Those answers identify the work that remains after the connector restarts, and what a successful recovery can actually restore.

Sources and scope

The analysis covered 1,965 public issues from Debezium, Confluent’s JDBC connector, Apache Kafka’s Connect framework, five Postgres logical decoding tools, and thirteen CDC products. Categories were assigned by keyword matching over issue titles and bodies; an issue could match more than one category. The offset and position category matched 539 issues, or 27 percent. These are matches in collected reports, not failure rates across deployments or measurements of typical recovery times.

Retention and recovery behavior depend on configuration. PostgreSQL permits unlimited slot-retained WAL by default, while MySQL binlog expiration is configurable. PostgreSQL 17 supports logical failover slots with synchronization requirements. Debezium documents resumable incremental snapshots and configuration-dependent snapshot and schema-history recovery.

Committed rebuilds destinations by replaying its own log of captured data, without rereading the source.