Well, I never use the word in terms of a vagina, unless I'm intentionally trying to be vulgar.

I feel it is perhaps the anatomical connotation that leads Americans to find it distasteful, but in the UK, I'd say 9/10 times the word is used it is not in reference to a body part, but merely a person.