Either my Vote belongs to me, or it does not.

Even now already your voting power is exhausting more and more with every upvote. And nevertheless your vote 'belongs' to you. After my model the returns would just diminish faster for accounts you have already upvoted recently. Of course even then your vote belongs still to you and not to me. :)

How about limiting the down vote ...

Naturally diminishing returns would also affect downvotes in the same way. So in case a whale is following you with downvotes, every single downvote would be weaker than the previous one. Then it would still be possible to flag content (for example plagiarism) if necessary, but less easy to hunt people ...

So you agree that quality is not subjective always. Apart from that also fictional stories can be evaluated concerning grammatical correctness, style, semantic richness ... I guess experienced reviewers will have various criteria to evaluate them. Of course there always remains the subjective part.