I would see

• a whitelist obviously
• a way to send push notifications (desktop browser),
• checkin my if the downvoted user is someone I follow or matches a list and send me a notification so I can manually review the downvote