Yes, but you gain it if you power up (or get delegation).

Delegation is something I didn't mention but the same thing. If someone delegates you 100,000 SP and you are low on VP, you will get the full voting power of that new delegation and it will not be crippled by your current voting power. When you lose it though, you lose that power as well.