Sort:  

I started downvoting them regularly when I stopped caring about my blog, reputation level, flags, etc.

It was kind of a relief to be released from the social pressure of conformity.