Except it doesn't just affect that piece of text. It harms the author.

Indeed this is true. Kind of feels anti-Non-Aggression Principle to me. Yet, most solutions I come up with could potentially be abused.

This is actually why I prefer no downvote (still think we need a report post function) as I see it as more of an attack, where I don't believe we really need attacks. I view it more like a market for content and ideas. If it is being approached like a stakeholder voting at a board of directors meetings that usually has other factors besides just market and yes/no might be needed. Yet even though we are stakeholders with our steem power (VESTS) I do think it could work without downvotes. There would still be problems with potentially people being unhappy someone was getting paid too much and FREE SPEECH would certainly allow them to be vocal about it, but they wouldn't be in a position to negatively attack someone other than with words. Currently this is very much not the case. A downvote can be used in a hostile attack method. We are counting on there being more good whales to offset this and that they will have time to commit to stay on top of this. Murphy's Law is screaming at me when I consider this.