Testing on Higher-End Hardware
- π€ Dave moves on to a 3970x Thread Ripper with an Nvidia 4080 GPU, which runs the 3.1 model quickly and utilizes the GPU.
- π He also tests the model on an M2 Mac Pro, which performs well and can allocate system RAM as video RAM.
- π Finally, he tests a 96-core Thread Ripper with an Nvidia 6000 Ada card, which struggles to run a massive 405 billion parameter model.
Conclusion
- π The size of the model and its complexity have a significant impact on performance, regardless of the hardware used.
- π Dave concludes that choosing the right model is crucial, and that even high-end hardware can be brought to its knees by large models.