The incident everyone remembers is the one where the pipeline went red at 3am. It is not the expensive one. The expensive one is the pipeline that stayed green.
What follows is a composite, assembled from patterns we see repeatedly. The shape is always the same: a change upstream that is correct on its own terms, a schema that absorbs it without complaint, and a number downstream that is quietly wrong for as long as it takes a human to get suspicious.
Day 0 — a correct change, upstream
A mobile team ships wallet payments. In the events they emit, payment_method gains a new value: wallet. This is a good change, reviewed by three people, and entirely correct.
The column is VARCHAR NOT NULL. It still is. Nothing downstream has any way to notice.
Day 0 to 19 — the shape moves
Over the next fortnight, wallet climbs from nothing to 31% of transactions in two markets. Every test in the warehouse continues to pass, because every test asserts things that are still true:
-- all of these still pass, every night
assert payment_method is not null
assert payment_method in (select code from dim_payment_method)
assert count(*) between 0.8 * yesterday and 1.3 * yesterdayThe third one is the interesting failure. Row counts stayed inside their band the entire time, because the total number of payments did not change. Only the composition did. Volume is the metric everyone monitors because it is the cheapest to collect, and it is very nearly the last thing to move.
The schema is a contract about types. It has never been a contract about meaning.
Day 19 — a human notices
Someone in finance asks why net revenue per transaction has drifted down about four percent in two markets and nowhere else. That is the first alert. It arrives as a question, in a thread, from a person who does not have access to the pipeline.
The investigation takes four people the better part of three weeks. Most of that time is not spent fixing anything. It is spent answering one question: when did this start? Without a per-column history, that question is archaeology — you reconstruct the past from whatever partitions have not aged out yet.
What would have caught it
Not a test. Nobody writes an assertion about the relative frequency of a categorical value, because there are thousands of such assertions to write and they all go stale.
What catches it is a baseline: a record of what the column normally looks like, scored on every run.
- Distribution shift would have flagged the category mix on day 11, when PSI crossed 0.55 — eight days before finance asked.
- Segment scoring would have named the two markets, rather than reporting a mild global change that is easy to dismiss.
- Replay would have answered when did this start in about a minute, instead of three weeks.
The uncomfortable part
Nobody in this story did anything wrong. The mobile team shipped a correct feature. The platform team wrote reasonable tests. The warehouse behaved exactly as designed.
That is what makes silent drift expensive: there is no bad actor and no bad decision to point at in the retro. There is only a gap between what your schema guarantees and what your numbers assume — and nothing standing in it.