It's not downvotes perse. The value you see below your post is potential payout and it is based on the reward pool which you share with everyone. It is also based on the value of Steem, which isn't always the same. That's why the potential payout is an estimate and it can fluctuate quite a lot. I've seen many a post go from $20 to $25 or from $20 to $15 throughout a week. I think this is what's happening to your post.

Downvotes (flags) are necessary to battle things like plagiarism. Yes, there are some 'flagging wars' between the big accounts from time to time, but most minnows who work hard, are fair and decent, and don't piss anyone off will not get flagged.