I spent quite some time about working out how to implement a fair and accurate reputation system, and one of the things that I worked out was that downvoting should not be permitted. Post a post instead saying how crap you think they are, see how that goes? Makes you realise what a dumb idea it is after all.

In my humble opinion, upvotes cost money, downvotes lower the poster on YOUR feed. Whoever you upvote, their downvotes lower the poster to the degree you upvote them on your feed. Downvoting should just be shunning, not actual punishment. Votes should be irrevocable.