You're spot on, of course, and I have long supported the use of downvotes to express a negative opinion; and changing the Flag to a Downvote as it used to be. Exclusively upvotes and positive reinforcement is unhealthy - there needs to be a diversity of opinion for any robust community.
That said, I'm on the fence now about flagging. It has proven to be deeply unpopular to the wider community, and we need to respect that. I doubt as a community we are ready for it just yet. It'll need a major perception shift and some UI changes that makes downvotes appear as normal and banal as they are on Reddit.
Having the 'flag' and 'downvote' combined into the same function is a large part of the problem. If we would be able to separate those two out and define "acceptable use" policies for each, I think that would sort out part of the issue.
There are always going to be users that get upset if/when part of their reward is removed due to a downvote though. I don't think there is any platform fix that can be made for that.
Yes, making a downvote a downvote and defining their use clearly is certainly worth a shot.
Of course, someone or the other will always be upset, but we can do better.
Based on the feedback that I've seen from the devs, this is something on their radar that they are planning 'eventually' but we will probably need to live with the current implementation for a while.