I think flagging should not be opposite as upvote. What we would need is actually upvote, downvote and flag. Let me explain why.

For example ... somebody writes a post on a football game he or she watched and I make a comment like

in my opinion team A played a better game, but has lost because referees made some mistakes ...

What happens is fans of team B will immediately attack you with some flagging, eventhough they don't actuall mean to flag you, it's just they don't agree with you. Seems flagging in this case could be pretty harsh punishment for making simple opinion.