I'm the furthest from a feminist (still respect women, just as I do men) and I agree with you. I think it has to do both with the fact that men tend to get more obvious arousals and with the keeping of women as "property" of their husbands. A man simply didn't want others to get curious in and possibly lust over his "rightful property". It was a monopoly that simply wasn't to be tested, even in thought.

(which as a supporter of property rights, I of course consider an insane system, yet for its time still understandable)