If we completely blocked content, we could not call ourselves censorship free. That is the down side of being censorship free on a blockchain level. That is why we need a downvote. It's not perfect but it is better than what Blurt has done. When they forked they created a coal system to replace the downvote. When you are suspected of doing wrong or not posting good enough content, you lose the upvote completely.

What we need to do is put more checks and balances in our downvote practices. Adding a community note is a good start. Also a button that a downvoter can trigger to tell the downvote bot that this post is a reward adjustment, dislike or illegal with a minimum allotted amount that a low quality post could earn.

I like your idea of the community needing to have enough votes for the community note to stay up.

If we were to give up the downvote, what would we replace it with? Because as it stands right now, content removal is decentralized. Anything we replace it with I fear would be more centralized.