I don't think the issue is with Rudy, doesn't matter if he's a pervert, these are character flaws and they don't make you hate a character on their own. What matters, however, is the presentation, when Rudy is rewarded with fanservice scenes that are blatantly drawn to look appealing, and the viewers are allowed to project themselves onto him and enjoy the moment, that's the problematic part.
That's just my subjective take on it though, personally I don't care much about fanservice scenes, I'm kinda numb at this point, nothing can make me feel uncomfortable, it's just I can't defend the writer (and the whole staff for that matter), they totally deserve to be called out.
In short, hate the writer, not the character.