That's the long tedious process of anarchistic living :). When people understand more, and share more of a common basis for reality, that difficulty gets reduced more and more... but it takes people to learn in order to understand how to operate and live this way.
Can the community create trusted agents to evaluate unfair flags and remove them? This would need to be mirrored in the backend code to allow such a possibility. Maybe some method to choose trusted people like escrow would use perhaps.
I have considered ideas about using approval voting weight (like witnesses) to pick people who have the "power to flag". It gets very tricky to scale and to hold people accountable, especially with incumbents.
Another method that has been adopted is random "meta moderation" similar to how slashdot.org does it.
Random jury pooling.