I agree entirely, Matt! Most of my following was built around commenting actively and basically engaging people on whatever they are writing about; I avoid "generic" commenting as it seems fake and disingenuous.
Sadly, I don't get out and about to comment as much as I would like to because I also place a high value on "curating" the comments on my own posts. Which sometimes means curating/replying to 40 comments a day... not much left over, after that. But I cut myself some slack; I have a full-time job and a part-time job, and this is just my "hobby," if you will.
When you are as popular a guy as you, you gotta make sacrifices. In your case it probably does make a lot more sense to respond to your current followers than to go out seeking new ones. Glad to hear you agree with this advice man, thanks for sharing your thoughts.