Yeah, but you lose the amount you downvote for, which could have been given to someone you considered worthy of it (OK maybe very few are worthy, but you've gotta upvote some people some time). And with the way people are voting these days, I think that amount that is downvoted is actually lost, I'm not even sure if the complete reward pool is being awarded on a daily basis.