I was once in such situation with a former friend, just different circumstances but similar accusation. It was bad and we eventually part ways. Those who mean to stay will stay and just like you said, " the people that actually care about you, won't put you at risk".
Yeah. Sometimes being caring and kind too much is not always good. They will get a chance to abuse you. Hays!