Most of the time, the reasons are personal and if it is about only social media, most of the time if I feel I am not interested into their niche content, music or anything,

Luckily I haven't had a personal reason so far to push the Unfollow button. It is more the interest. But I didn't really thought about how it may be perceived as for me it is clear why.

I don't keep track of who unfollows me, I always assume who does is free to do so and has its reason.

keep my circle small

Just as in real life. The closer, the smaller the circle.