Part 2/8:
To delve deeper into how this works, researchers at Google DeepMind have analyzed the inner workings of these models. While a full understanding remains elusive, their findings indicate that certain facts seem to reside in a specific part of the network known as multi-layer perceptrons (MLPs). With the foundational knowledge of transformers— the architecture that underpins these models— we can explore how MLPs contribute to knowledge storage.
Each input token, representing chunks of text, is transformed into high-dimensional vectors. As the model processes these vectors, they interact through various operations, predominantly attention mechanisms and MLPs, which are crucial for enriching the vectors with contextual meaning.