Your CDC pipeline says RUNNING. Your data stopped moving.

In my analysis of 1,965 public issues, I found that 17 percent of issues matched a category covering hangs, stalls, and silent failures. It is one of only two categories that shows up in at least five percent of issues in all five source groups.

Some of those reports describe an insidious failure: the pipeline was reporting that it was healthy while no data was actually moving. In some cases, operators discovered the problem days later when they noticed missing downstream data, without having received an alert.

This piece explains how this happens, and what to monitor alongside task state so you can detect it.

What RUNNING actually measures

CDC frameworks commonly expose task states with names like RUNNING, PAUSED, and FAILED. When you see one of these states, it is easy to assume they describe what is happening to your data, but task state alone does not tell you whether changes are reaching the destination.

The exact meaning of RUNNING depends on the framework. A task can remain RUNNING while retrying a failed operation or waiting for a read to return. That does not tell you whether the position has advanced, whether the source has produced changes the pipeline has not yet consumed, or whether anything reached the destination. The framework may expose different metrics for those things without using them to determine task state.

That is how a task can report RUNNING even when it has stopped delivering data.

Seven ways delivery can fail without a useful status signal

1. The retry loop. An error (a timeout, a connection reset, or a throttled sink) triggers a retry. The task can remain RUNNING while those attempts fail. The reports include a retry limit that resets when it is exhausted, allowing the pipeline to keep retrying even when the underlying problem cannot resolve itself. The outcome is that the task is still attempting work, but it is not making progress.

2. The source stopped talking. The connector keeps an open connection to the source database so it can receive changes as they happen. A source restart or network interruption can leave the connector waiting for data on a broken connection. If timeouts or keepalives do not detect the failure, a read can wait indefinitely. The thread will still be alive and waiting, but no changes will arrive.

3. The position is past the head, or before the horizon. A connector saves its position so it knows where to resume. That position can become unusable in either of these two ways: it can point beyond what the source currently has, or it can refer to changes the source has already deleted. Depending on the connector, this can produce empty reads or repeated errors. The process may keep running even though it cannot make progress.

4. The snapshot with no progress counter. A pipeline reporting SNAPSHOTTING tells you what it is doing, but not whether it is getting anywhere. The bug reports describe snapshots that stopped making progress and snapshots that started over after an error. Without metrics showing how much has been copied and whether the scan has restarted, those can look the same as a snapshot that is working.

5. The write fails, but the position advances. If a write fails but the error does not percolate back to the pipeline, the pipeline may advance its saved position as though the write succeeded. Consumer lag may read zero and the pipeline may keep moving, but there is data missing from the destination.

6. Backpressure with no signal. Changes arrive faster than they can be delivered. The queue holding them fills up, so the connector has to stop reading new changes from the source. The task can still report RUNNING while it waits. If that also prevents a replication slot from releasing old WAL, the source keeps accumulating log data and can run out of disk. A retention limit can protect the disk, but deleting changes the connector still needs creates a recovery problem.

7. Partial death. One stream stops delivering changes while others continue. The bug reports describe streams that failed when schemas were changed. In some of these cases, the operator did not receive the notifications they expected. If you only watch the pipeline’s total throughput, the activity in the other streams can hide the failure. Some data is moving, but that does not mean every table is keeping up.

What a real signal needs

A useful status endpoint should show progress for each source and make it clear when a worker needs intervention:

• Phase. Is the source snapshotting or streaming? Those involve different work, so progress needs to be interpreted differently.
• Snapshot progress. Is the snapshot advancing? A phase label alone cannot distinguish a slow scan from one that has stopped.
• Position. Where has the pipeline reached? Watching the position over time shows whether it is advancing.
• Lag. How far behind is it? An advancing position does not tell you whether the pipeline is keeping up.
• A caught-up flag. Has it reached the source’s current position? This distinguishes being caught up from merely making progress.
• A parked state. Has the worker given up? A worker that requires intervention should say so instead of continuing to appear RUNNING.

Alert when lag exceeds an acceptable threshold, a snapshot stops advancing, or a worker gives up. If the position stops advancing, check whether the source has new changes to send. These signals make stalled capture visible, but you should still check that the destination has the correct data.

The analysis covered Debezium, Confluent’s JDBC connector, Apache Kafka’s Connect framework, five Postgres logical decoding tools, and thirteen CDC products. Categories were assigned by keyword matching over issue titles and bodies, and an issue could match more than one category. The seven mechanisms above draw on individual reports; they are not seven separately counted categories.

Committed’s status endpoint returns phase, snapshot progress, position, lag and a caught-up flag for every source. A worker that has given up is shown as parked rather than running.