Please leave, you're scaring away customers

What does racism really mean? That's what I'm trying to figure out using a series of thought experiments. This is the second one.

The Experiment

My father used to run a small burger stand when I was a kid. Sometimes, the local biker gang would come around. Some would order, others would not. All of them would sit there, though. They were scary-looking, but, as far I can remember, they weren't much trouble.

My father noticed that a lot of potential customers would avoid his burger stand, when the bikers were around. So he'd ask them to leave after ordering. I don't know how they took it, but I can't recall there being an issue. They left. Or perhaps they didn't.

What intrigues me is that no one I've told this to has had any qualms about asking bikers to leave because they look scary.

But what if the bikers were not bikers, but instead members of a race that looked 'scary' too?

Would that be any different? If so, how?