Once your downvote power goes to zero, it is supposed to start using your regular vote power for downvoting. If you find that it doesn't, that's a bug, please carefully document the conditions and report it.