I analyzed 1,965 CDC bug reports. Here is what actually breaks, ranked.

Most writing about change data capture is about setting it up. Almost none of it is about how it fails once it is running, which is odd, because the failures are public. Every CDC tool of any size has a bug tracker, and the people operating those tools file bug reports about exactly what went wrong.

So I ran analysis over a bunch of CDC tools. I analyzed 1,965 issues, from five places: Debezium’s GitHub tracker (897, every issue typed as a bug, including what migrated over from its old JIRA), Confluent’s JDBC connector (99 issues filed since 2023), Apache Kafka’s Connect framework JIRA (38 bugs filed in the last two years), five Postgres logical decoding tools: wal2json, pglogical, spock, pgcopydb and pglogrepl (130 issues in the last two years), and the public bug trackers of thirteen CDC products, from Airbyte and RisingWave to PeerDB and Sequin (I only counted the CDC issues and it came out to 801 issues in the last two years). During analysis each issue was categorized, the categorization rules were spot-checked against samples from every bug tracker, and then the issues were counted. An issue can be in more than one category, so the percentages below do not add to 100.

Here are the rankings, with an explanation of each entry:

1. Losing your place in the log (27 percent)

A CDC pipeline is a position in an ordered log: an LSN in Postgres, a binlog coordinate or GTID in MySQL, a resume token in MongoDB, a change tracking version in SQL Server. Everything the pipeline knows about “where am I” is that one value, and more than a quarter of all reports are about it.

The failures come in three shapes.

1. The position is not advancing: the connector keeps reading but never confirms (acks), so the source keeps every byte of log since the last confirmation and the disk fills.
2. The position is stored wrong: the offset is committed before the write it describes has landed, so a crash loses the rows between the two and nothing ever reports that it has happened.
3. The position is unusable: the source deleted the part of the log the position referred to, or a failover moved the pipeline to a replica where that position means nothing, so the only way to fix this (without losing data) is to perform a full snapshot.

It is the largest category in three of the five bug trackers. It is also one of only two categories that makes up at least five percent of the total bugs in each individual bug tracker.

2. Reporting healthy while stalled (17 percent)

The second largest category is the pipeline that reports “RUNNING” while nothing is moving. If your retry loop is retrying indefinitely, it looks like a healthy task from the outside. If the source silently dropped the connection the pipeline will wait on a read that will never return. If you have a polling loop that is looking for a position that is past the head of the log it will return empty forever and empty looks exactly the same as a quiet day.

What makes this category different from the others is how it is discovered. The reports almost never say an alert fired. They say a table downstream stopped changing and somebody eventually noticed, with the few that say how long putting it at hours, days, and in one case over a week.

3. Schema changes (15 percent)

Someone runs a migration on the source and the pipeline breaks, in one of three ways.

1. The engine keeps its own history of the source schema and that history drifts from reality (usually after a restart or a re-snapshot). Every event after the drift is decoded against the wrong shape.
2. The new column reaches the pipeline but the sink cannot alter the destination table, so events pile up or get written with the column missing.
3. A type change that the source considers routine (widening an integer, changing precision) becomes a hard error at the sink because the destination column will not accept it.

Nearly a fifth of Debezium’s reports are schema related. It is the failure mode with the highest ratio of “this happened during a normal workday” to “this happened during an incident.”

4. The replication slot fills the disk (9 percent)

A Postgres-specific consequence of category one. A position that stops advancing holds the write-ahead log open. This means the source keeps every byte since the last confirmation, and the disk fills.

This is 15 percent of the vendors’ CDC issues and 8 percent of the Postgres tooling’s. It is hard to triage correctly because the error shows up as “out of disk” on the source, not as an error in the pipeline.

Postgres 13 and later can cap how much log a slot may hold and invalidate the slot past the cap, but the cap is off by default. Setting it protects the disk by converting this failure into category one.

5. Timestamps (8 percent)

Microseconds get truncated to milliseconds, a naive timestamp assumed to be UTC when the source session was not, a high-precision SQL Server type mapped to something narrower at the sink. The rows arrive in the sink, the counts match, but the values are quietly wrong by a few hours or a few microseconds.

The pipeline will not flag any of this because it is doing what you told it to do. This is most likely to be found by someone checking that what is coming out of the pipeline is the same as what entered it.

6. Data loss and drift (5 percent)

Data loss and drift (missing rows, duplicated rows, counts that do not match) is almost certainly undercounted, for the same reason I talk about in “What the ranking cannot show you”.

7. Forced full re-sync (4 percent)

Forced full re-sync is only 3 percent of Debezium’s reports, but it is 7 percent of the vendors’ (more than double).

I don’t think the users of products encounter this problem more often. I think the same failure gets filed under a different name. On Debezium’s tracker the reporter is the engineer running the engine, and a lost position is filed as a lost position and they also mention the slot being invalidated, offsets missing, or the log being purged.

On a vendor’s tracker the reporter is a user of the product. Their product’s recommended fix is typically to re-snapshot everything again, so they do that. Even though two thirds of the vendors’ re-sync reports use “reset” type vocabulary, more than half of them additionally mention the lost position in their report. This category and category one may be the same failure counted twice: once as a cause and once as a cure.

8. Memory (2 percent)

Snapshots of large tables (a single wide binary column can do it), a schema history that grows without bound, a leak that shows up on restart or after a long stream. Out of memory isn’t what you would worry about first, but it does happen.

What the ranking cannot show you

A bug tracker is a collection of bugs representing people who noticed a problem AND took the time to report it. By definition the two silent categories, “Data loss and drift” and “Reporting healthy while stalled”, only get noticed when someone downstream happens to compare. This means the percentages in these two silent categories are floors, not accurate measures.

The loud categories (crashes, a full disk, a schema error that stops the task) are probably counted fairly because the errors that the pipeline tells you about are the easiest to notice. The silent ones are harder to notice so the categories may be ranked lower than they should be.

One more number. Among reports filed by users (defined as people with no commits in the project and no membership in its organization (Kafka’s JIRA doesn’t record this, so we count every reporter)), every tracker had a share that never received a single reply.

1. Debezium was 33%
2. JDBC connector was 52%
3. Kafka Connect was 29%
4. Postgres tooling was 32%
5. Vendors were 15%

The rate drops in the vendors’ bug trackers, and I hypothesize this is because they pay someone to answer reports. This list is not meant as a criticism of the maintainers (maintainers are few and reporters are many). It is a description of what “we run it ourselves” actually means when something goes wrong.

What to ask of a pipeline

If I were evaluating a CDC setup, existing or proposed, the ranking suggests asking five questions, in order:

1. Where is the position stored, and is it committed after the write it describes, or before?
2. What does the pipeline report when it is alive but not moving, and is that different from what it reports when it is moving?
3. Which of the three failures above does the setup choose when there is a schema change?
4. If the position is lost, what is the recovery path, and does it require a re-sync from the source database?
5. When it breaks, is anyone paid to look into it?

I am building Committed in part because of this list. The status endpoint and the log-first recovery features are direct answers to questions 2 and 4.