Most of the time it just easier to ignore trolls and spammers... likewise with the downvotes that don't have any power behind them. It is a touch harder to ignore the big ones though... and therein lies the problem... unless you have a big friend to "heal" the damage... then there is really no consequence and you just take the hit.

This is not something that can scale well... as the "good" guys can't be everywhere... and it can be hard to drop into a situation and know what the complexity is behind it all... thus, there needs to be some way of making actions have consequence.