It's mostly because of one's capability to pay back the money in time.
Not everyone has enough money to lend out. Some saves a part of their savings for emergency purposes. We dont know what will happen in the days to come. One could get into serious trouble, an accident or might need the money for something totally unexpected and that is why peoplr are reluctant to lend out money to the poor because they might not pay in time.
There is also the issue of what will he use it for. I am not generalizing everyone, but some borrow money just to satisfy their addiction whether it be drugs, gambling or even women.