There is no 'have to' only you wanting something or not.
I also love sports a lot and when I am not doing photography I am most likely doing something sports related. Shifting between different passions is healthy for the longer term I think
Looking forward to what kind of photos you will be bringing, what kind of photography do you do for a living?