A few bad apples (forgive the pun) spoil the fun for everyone else.
I love the concept. In America there are these little "libraries" (more like glorified mailboxes) that people can swap books at, unattended. But even those have their critics and are prone to abuse.
We have a couple of those -- "Little Free Library" -- in my neighborhood. Take a book, leave a book. I've found some good reads, and donated things to clear shelf space at home. I'm a big fan, haven't heard any criticism or complains about them here.
oh I like that idea of a little honesty book swap library. I have seen a few book swaps in Australia but they are usually located indoors, like a laundromat or something.