I am sorry to say but it sounds like you have them in your walls.
I do too!

My only solution has been to make offering to the ants.
I throw all compost able material nearby in my backyard, and I edge it further away from the house slowly but surely.
They've went away entirely.

If you need some help, I've become an ant expert, so if you tell me the type of ant (color, shape, size) I might be able to help give you an insight into how they operate.
If they're black-ants they're sane, rational agents.
If they're red ants they're aggressive but relatively rational.
If they're "ghost"-ants, you need to get yourself infested with black ants. Make offering and the black ants will come. They will chop those tiny ghost ants up and stay in their own territory where the offerings take place.
I'm serious.