I agree. People are not going to waste their voting power to downvote self-voters, it's not going to happen unless they have real incentives to do so.
Like you said there is no way to determine if the self-voters deserve a downvote, this is going to create a very toxic environnement if people start to downvote left , right and center.