I think you're right. I think if you said 'White lives matter.' folks would come out of the woodwork to call you a racist. Well, that means saying 'Black lives matter.' is racist.
Our lives matter, and only racists think otherwise, or want to be the only race that matters.