I definitely can relate to this ! I completely changed my tactic a few months back and removed all the people I don't usually comment on or have any interaction with - if that means they will unfollow me later I have no problem with that.

What I want to do is to have meaningful correspondence with people that I'm interested in and that are interested in. I truly think you mention one of the most important things on here - it's a great advice