You get a warning saying "Removing your vote will reset your curation rewards for this post." which makes sense - no vote, no reward. And if you don't vote again you won't get a reward. But if you do vote again you'll get a reward based on the time you re-voted. So you don't lose anything unless you don't re-vote.