The Anakites are also said to be descendants of the Nephilim. The way I figure it, they original Nephilim were giants, but since the angels originally procreated with women, their offspring did the same thing. Over time less and less angel "DNA" was in their blood and more human, so their change in size and stature would have been caused by this. This would have led to the eventual "extinction" of the giants. There may still be traces of their genetic make up within modern humans, but it's been mixed with it for so long they're completely indistinguishable.