Unless his followers have more rep than you, they aren't able to affect your reputation. Most of them seem to have very low rep.