The self-voting has been an endless debate.

It will continue to be so until something changes. I'm afraid flagging is not enough for this problem.

Unfortunately, it is a mechanism that is indiscriminate towards any of the criteria you noted and there are not enough people flagging the problem of content that is overvalued due to excessive vote buying and self voting. It's just the nature of the beast right now.