Hate it. I work from home and people, mostly relatives, think that because I'm home, it's OK to just stop by to visit. I don't just stop by your workplace and expect you to stop working and spend an hour or so visiting with me. Why would you stop by mine.
For one thing, it's just plain rude. Why would you think that an unannounced visit to someones home would ever be appropriate. You don't visit unless you are invited. If you are going to be in the area, call as far ahead as you can and ask the person if they have time for you to stop by for a SHORT visit. If they say yes, then keep it short.
I agree with what you say. I am a stay at home mom at the moment and people think because I'm home it's okay to come by. I feel like I'm judged when relatives stop by. If my house has a little mess then I get looks or comments. Relatives are a pain in the butt.