We already have laws concerning squatter's rights in the USA. There are laws about something called "adverse possession". Depending on the locality, a person can claim property if they have been living on it openly for 3 to 30 years. https://budgeting.thenest.com/claim-abandoned-real-estate-32741.html
Would the "private protection agency" be like a mafia?
exactly, or you could call it a "cartel" or "street gang". You don't even have to live there, you just have to use the property for the specified period without permission. If they give you permission at any time before you acquire the property your claim disappears. You could go mow a field and take a picture every year and as long as you don't have the property owners permission then you can take it by adverse possession.