Your answer assumes coordination. You say "the rest of the society" would have to decide to do something, but because there's no government to coordinate them, there is no effective "society" that can make decisions as a unit. There are only individuals, and individuals are vulnerable to coercion.