People don't only downvote because you've done something "wrong" but also because they disagree with the amount of rewards you're getting. People upvote when they think the post deserves more rewards and downvote when they think it has received to much.