It helps to be fit and skinny, though I am neither. There is some climbing involved if you are determined to get into places.
Last Sunday I was flat on my belly scrambling under a gate, delicately trying to maneuver in an old mansion with little floor left (a ten-foot drop awaited me if I slipped), and climbing through a stairwell as someone had barred the door leaving me little options.
Other places, you can walk right in unhindered. These are the ones you might want to look for initially.