The Fermi Paradox is indeed a scary question.

I personally like this X-Com quote:
Either we are alone in the universe, or we are not.
Each, is equally terrifying.