We need to separate flags from downvotes. Simple as that. Although if it was simple, then why is it not done already?

At least the Busy platform seems to be working on adressing these issues. So far they use "like" and "dislike" buttons, so the flag is gone. Hopefully they will do even more in a later stage.