According to @dan:

Those with weaker reputations are unable to harm the reputation of those with strong reputation.

So you don't need to worry!