I lost my fear after I found the article of Bronnie Ware called "The Top Five Regrets of the Dying" which are:

1. I wish I'd had the courage to live a life true to myself, not the life others expected of me.
2. I wish I hadn't worked so hard.
3. I wish I'd had the courage to express my feelings.
4. I wish I had stayed in touch with my friends.
5. I wish that I had let myself be happier.

Especially 1, 2 and 5 touched me - this was where my change to downshifting started.