This doesn't de-penalize that. When you vote and unvote, you don't get your vote power back.

This was a different issue: once you vote on a post, unvote, then revote, your revote didn't give you any curation rewards. It is strange and unexpected behavior.