There's this fatigue or should I call it stress that suddenly envelopes you when you're not doing that thing you are supposed to.

Yes, at that time the mind isn't at rest. It just keeps wandering back to that thing till you eventually decide to get it done.

Its always best to take responsibility sooner rather than later, else things could go bad really quickly.