I think downvoting only becomes a perennial powder keg if the inequality in the reward distribution remains as high as it is now. In a less extreme scenario where neither a single upvote nor a single downvote carry such an excessive weight people will probably be rather meh about it.