Yeah, this is a part of how our nervous system works. When we have focused attention, the patterns in our associative system are rigid. When we are waking up or falling asleep or daydreaming, the patterns become less rigid, so we make more novel associations.
It appears that the parts of the brain that are very active when we have focused attention are inhibiting the parts that are necessary for making very novel connections. By actively working on something, it appears that we prevent truly creative solutions from coming to us. (I mean "coming to us" metaphorically, since there are subcortical parts of the brain involved in making the novel connections, and when the information is carried to the neocortex, now it's in our conscious awareness and we get a sense of "it came to me from out of nowhere").
Wow, you just explained in details a phenomenon I've been experiencing without explanation.
Could it be same reason why I improve drastically after sleeping over a new skill?
Let me illustrate, the day I climbed a bicycle, it was woeful, I couldn't even direct my guess neither could I pedal but the next day it was all smooth.
I experienced something similar learning to play the violin.
And most of these times I dreamt doing the same thing.
Glad if you found my explanation useful.
Yeah, the solidifying of skills after sleeping would be a different physiological mechanism. It has to do with the connections between the neurons. When you learn something new, you are building new connections in your brain. During the night, it appears that unused connections get trimmed. So you're left with what you practice. If you practice the wrong things, you'll end up building the wrong habits (wrong neural pathways). But if you practice correctly (you repeat the thing the way you'd like to do it, not just any way), then you build the neural pathways you want. And those remain, due to them being utilized, while the other connections become weaker and disintegrate.
At the same time, with practice, it appears that a pathway becomes more efficient. So that's probably why it takes less and less effort and conscious attention for us to do something after we've practiced it a lot. It becomes automated/habitual. This is great because it allows us to build complex skills that consist of lots and lots of small movements (like walking, for instance), and we can even do other things at the same time (like walking AND talking). However, a downside is that now we have this habit we're somewhat stuck with. If we find a better way to do the thing (a better way to walk, to type on the computer, etc.), then we have to rebuild new pathways from scratch, while avoiding repeating/using the existing pathways.
Here is a TED talk that goes into some of the physiological mechanisms underpinning learning: