Things are how they are. In an ideal world my opinion is that you would be correct. However, I will always express my opinion against liberalizing downvotes. Human beings are, for the most part, emotional beings. Even if a downvote is meant positively, the reaction will be negative.