Reading a bunch of the comments, it appears like they are mostly opinions without much thought behind them. It is a very complex topic and people use down-votes differently than they use up-votes, so it isn't a simple symmetry. Honestly, the whole concept of up-votes and down-votes lies on some very questionable assumptions of group cohesion. For instance, the popularity of a movie tends to be a counter-indicator of it being a persistent cult classic.

As a society moves further away from 'healthy' 'unhealthy' things rise in popularity and 'healthy' things decline. Substitute X & ~X for 'healthy', 'unhealthy' to get the the generalized principle for any dipole category. I think when people publicly classify content in some way, they should be publicly classifying themselves as well. The current system doesn't have that aspect. Going back to the movie popularity example, imagine how useful it would be if you could factor out all of the votes from people who think car crashes are the most important aspects of movies.

Another example: Imagine having a popularity contest for fruit. If strawberry wins and it also happens to be your personal favorite, you would probably see nice effects from that result. If your favorite is actually orange, you may see some kind of negative results. Most people may actually prefer something other than the popular fruit. What does a popularity contest achieve? Is it intended to validate the people who like strawberries? Provide a social pressure to people who like oranges to conform?

This isn't a dissertation, so that is enough to hint at the serious problems underlying the popularity assumption. I personally don't think any particularly interesting results are going to come without looking at the bigger meaning context. I.e. it doesn't really matter what you do with down-votes because it will still be fundamentally flawed.