That's very true : we draw inspiration in the design of artificial neural networks from biological inspiration. This is what McCullogh & Pitts tried to do when they formalised the first version of an artifial neuron [1]. Like you mentioned, the current advances in NNs are driven by tried to better understand the statistical properties of this function approximators in order to make them better (and faster / more efficient) because we still need to run these things :D
Nevertheless, we do in fact suppose that some of the behaviour we have observed in neural networks' intermediate representations can help us understand the way our brain models the information at different levels of abstraction. The paper mentions this at the end of the discussion section :
We conjecture that a stimulus may be processed by binding neurons into cliques of increasingly higher dimension, as a specific class of cell assemblies, possibly to represent features of the stimulus (Hebb, 1949 [2]; Braitenberg, 1978 [3]), and by binding these cliques into cavities of increasing complexity, possibly to represent the associations between the features (Willshaw et al., 1969 [4]; Engel and Singer, 2001 [5]; Knoblauch et al., 2009 [6]).
This is very similar to what is observed in ANN, most notably CNNs (because images are fun to look at :p). I've linked the appropriate papers below if you're curious to check this out !
Thanks for the comments @manfredcml !
[1] http://www.mind.ilstu.edu/curriculum/mcp_neurons/mcp_neuron_1.php
[2] Hebb, D. (1949). The Organization of Behaviour. New York, NY: Wiley & Sons.
[3] Braitenberg, V. (1978). “Cell assemblies in the cerebral cortex,” in Theoretical Approaches to Complex Systems, eds R. Heim and G. Palm (Berlin; Heidelberg: Springer), 171–188. Available online at: http://www.springer.com/cn/book/9783540087571
[4] Willshaw, D. J., Buneman, O. P., and Longuet-Higgins, H. C. (1969). Non-holographic associative memory. Nature 222, 960–962.
[5] Engel, A. K., and Singer, W. (2001). Temporal binding and the neural correlates of sensory awareness. Trends Cogn. Sci. 5, 16–25. doi: 10.1016/S1364-6613(00)01568-0
[6] Knoblauch, A., Palm, G., and Sommer, F. T. (2009). Memory capacities for synaptic and structural plasticity. Neural Comput. 22, 289–341. doi: 10.1162/neco.2009.08-07-588