Your reputation is a statistic awarded to you based on the votes you receive. When someone votes you, you receive reputation points, when someone flags you you lose reputation points. The amount of reputation they can give or take away is relative to both the amount of SP they have as well as their reputation. So someone with a high reputation and a lot of SP can increase someones reputation or decrease alot depending on whether they vote or flag.