So it's very much a Darwinian process.

Exactly. There was a whole lot of natural/cultural selection that bred for attack/defense, rather than community. As you said - martial culture.

They were just simply better at killing people, and more resistant to diseases bought by invaders (just ask the poor Incas...).

This last one about the "resistant to disease" is a big one, isn't it? The more homogenous a community genetically and across exposure to disease, the more prone to being wiped out.