Part 7/15:

The Role of Knowledge and Memory

A crucial insight is distinguishing knowledge (static information stored in weights) versus working memory (dynamic, accessible context). Karpathy compares the neural network's "hazy recollection" in the weights—compressed, noisy, and often outdated—with real-time context and working memory that is directly accessible and flexible.

He advocates for reducing the reliance on stored knowledge in favor of maintaining a dynamic "cognitive core"—algorithms, strategies, and reasoning capabilities—and minimizing the “silently collapsed” knowledge that encodes too much about the past, which hampers adaptation and continual learning.