Good point - beyond the actual impact of the downvote there is the impact on people's pride which can elicit a rather irrational response. One could ignore that - but hostile reactions do set and poison the tone on a platform.
So, yes, you might be right that downvotes are a bad idea even when their monetary impact is small.
If I saw down votes as solving something that couldn't be handled other ways I'd be all for it. Yet I decouple down votes from the reporting plagiarism, spam, abusive posts. Those things I consider as needing to be flagged or down voting.
I've been fortunate. I have not had too many down votes, and the few I did were on posts weeks after I wrote them. I can be devil's advocate and the opposite of many people I have spoken to here, but I do try to keep it civil. So do they. I love that.