Philosophical Question about People in your Environment

There’s a philosophical question I asked myself today. There are many people you truly love. You value them and you really like them. In return, they love you too. But somehow, there are some people you truly love but hate spending time with. They might be friends from school or work, parents, or relatives. There are probably also several reasons why you dislike spending time with them. They are dragging you down to their mentality. You both don’t agree on basic topics. Intellectually you cannot have a real conversation. They are only complaining about everything. Talking bad words about other people. Don’t support you on what you are doing. Whatever the reason might be. I asked myself whether it is worth to spend time with people you truly love – and with truly I mean truly truly – but who drag you mentally and intellectually down. What is your opinion?