Just as you feel I don't think it should be scrap completely so that the community will be not be spam,but their should be a kind of restrictions for those that downvote without any genuine reason is of doing that..
As I heard some even down voted just because the dislike you post I don't think that is a good thing
Downvotes should still exist but users should be cautioned from using downvotes as a tool to attack some users..