I can speak from experience on this point - one of my posts made No. 1 on the front page and was worth about $8k. Shortly before payout a bunch of downvotes showed up and knocked the payout down to $5k. I was pissed! Not because they downvoted, but because they downvoted without commenting and telling me why they downvoted. I think they just wanted the reward reduced, which is not an action that should be condoned.

So here's my thoughts - downvotes only get applied if they are accompanied by a comment that is upvoted by someone other than the OP and the down voter. Boom!