I don't even let things get that far. If I don't recognize the number, I don't answer the phone. If it's important, they can leave a message.
Same with the front door. If I'm not expecting anyone and I don't know who it is, I'm not answering the door.