AIVersee AssistantOnline — answers instantly
Want to run Llama, Qwen or Mistral on your own GPU? Pick the model size and quantization — get the real VRAM you need, weights + KV cache + overhead.
Estimates for inference (running the model), not training — training needs roughly 4–8× more. KV cache figures assume modern grouped-query attention; exact numbers vary by model architecture. If the total is close to a GPU's memory, choose the next size up or drop the context length.
Weights = parameters × bytes per parameter. A 7B model at FP16 needs ~14 GB; quantized to INT4 it drops to ~3.5 GB with a small quality cost — that is why 4-bit is the sweet spot for home hardware. On top of weights you need the KV cache, which grows with context length and batch size, plus ~15% runtime overhead. Add all three and you have the GPU you need.
About 14 GB at FP16, 7 GB at INT8, or roughly 4.5–5 GB total at INT4 including KV cache and overhead — so an 8 GB GPU runs 7B models comfortably at 4-bit.
At INT4 the weights alone are ~35 GB, so no — you need around 40 GB total, meaning a 48 GB card or two 24 GB GPUs. At 8K context, a single RTX 4090 tops out around the 32B class at INT4.
Yes. The KV cache grows linearly with context: on some models, jumping from 8K to 128K context adds more memory than the quantized weights themselves.
For chat and general use, modern INT4 methods (GPTQ, AWQ, GGUF Q4_K) are close to FP16 for most people. INT8 is nearly indistinguishable. INT2 is where quality drops sharply.