You're right—model size (parameter count) heavily influences LLM capabilities like nuanced communication, sarcasm detection, and natural language flow. Larger models (e.g., 70B+ params) capture more complex patterns but demand massive compute (GPUs/TPUs) and energy. Smaller ones trade depth for efficiency, which suits Rafiki 1.0's real-time Threads setup.
For your in-house build: Start with open-source like Llama 3 (8B-70B) on consumer hardware (e.g., RTX 4090 for 7B). Scale via quantization (e.g., 4-bit) to fit RAM. Tools like Ollama or Hugging Face simplify setup. Check recent benchmarks on model scaling impacts for hardware recs. Good luck—sounds fun!
Looking at a RTX8000 and 128GB of system RAM for my machine. I should be able to handle a 4 bit quantized 70B model or a full 30B+ model.
Yes, an RTX 8000 (48GB VRAM) with 128GB system RAM can handle a 4-bit quantized 70B model like Llama 3—VRAM usage drops to ~35-40GB, leaving room for context and inference. It supports full 30B+ models (e.g., unquantized Mistral 30B) comfortably at ~24GB VRAM.
For optimal performance: Use llama.cpp or Ollama for quantization. Offload layers to RAM if needed for longer contexts. Power draw is high (~300W GPU), so ensure good cooling/PSU.
From recent guides, RTX 40-series (similar VRAM) run 70B Q4 efficiently on similar setups. Test with smaller models first to benchmark.
I am using Ollama