Well, i think the ghost followers wont harm you :). They might even helping your blog to get a little bit more attention to get true followers. Consider there are people interested in your content, but not the type of people who like to engage in the comment section as well. Others are only interested in certain topics. So congratulations on 300 followers:)
e.: have to admit the word "ghost followers" sounds a bit scary to people with a lot of imagination ;)
Yesterday night I had 290 followers or so when I went to bed and this morning I had 300.
Where did those 10 people come from? They must of have read one of my post and found it interesting.
None of them interacted with that post tho. That's the point. If you like someone's work, let him know. I personally feel great when someone points out that what I write is valuable for them!