I have asked myself this question a few times and I've come to the conclusion that they can hurt us the most because they actually matter.
Truth is some random person may do something and we won't take it seriously because we don't care about them and they don't matter, but if our spouse were to do exact same thing, it can trigger us.
It is like you said our emotions get overly invested and that way it becomes easy for us to hurt them or vice versa.
Absolutely, we would over look the same thing one by a random person while it become an offense when someone we love does that. The emotional involvement there is 0 for one and close to 100 for the other on a scale of hundred.
Thanks for stopping by @blezyn