I think that would make perfect sense! As a user I see value in these lists because it helps you know when you might want to second guess your votes. When the flag is used on things I obviously disagree with it makes it harder to trust all of the rest of the good guy's judgments since their mistakes taint the authenticity of the badge for those not using it on an old lady's religious beliefs.