From my understanding it has to have a certain number of ref votes before rewards are removed. If you see something you don't think is right just hit it and stick it in the replay room. 2.5 downvotes a day gives you the power to remove 15 000 tokens a day or so which is more than 20 Steem.