Yeah, it's one of those things. I mean, math is hard for everybody and the better you are at it the harder it gets. Personally, I have a a BSc in PMAT and dropped out of my PhD, which probably puts me above average, but I doubt I have more ability than anyone, just that when problems get stuck in my brain I can't let go of them.
That's all there is to be great at math, no different then becoming an athlete. Just get addicted to that pain!
I totally agree with you @blendletan :))