If you actually try to use your feed, then yes, you are best served leaving the follower side mostly unused for now.

However, reputation filters are coming which may help a lot.