In theory there is nothing stopping someone from creating a hardware wallet that's 100% under their control.
It would just be very difficult and expensive given the current tech stack.
Easier to outsource that work to a centralized agent at the moment.