Well, certainly an infrastructure would be needed for the deployment of the network, and to be honest I can't think of a way to prevent (decentralize) that fact. But hey, I'd say one step at a time; as you mentioned before, most ISP providers can do as they please with our information - we could start from there, a decentralized organization, by its own nature, would not find this practice ethical.
As for the physical part you mention, who knows, maybe in a near future the power to do this would not be centered on a particular organization, perhaps a lot of people could have the knowledge to do it, and the internet provider would only give you access to the service, but not the installation? I know it sounds pretty unrealistic, but maybe it is feasible in some way.