I'd say even if you don't feel like saying no to someone, because that is a very personal "no", you have the option to just continue without them. There will ever exist people that take part in our lifes. Not for long, but for a quiet good time. When this time ends we can just go on and be glad for the shared moments we had. There is nothing wrong in saying: "Hey, meeting you was nice. I see you will pull out to the left and I decided to pull out to the right one. Maybe we meet again. Have a good time."