Roughly 40GB. At 4-bit quantization. On a single high-end GPU. There. That is the number everyone is actually asking for, so it is up front: how much VRAM for 70B model, answered. That one number decides whether your project is feasible or fantasy, so I would rather you had it in the first ten seconds.
Longer version, because precision is the whole game. Full precision wants around 70GB, which is why almost nobody runs a 70B that way. At 4-bit you land near 40GB, and the quality stays surprisingly close to the original for most tasks. Go lower and the memory keeps shrinking while the quality cost grows. No free lunch. Just a menu, and you pick your tradeoff.
Context length matters too. Those VRAM figures assume a sane context window. Crank the context up and the KV cache eats more memory. Long-document work on a 70B needs headroom past the base number. Do not size to the exact minimum unless you enjoy out-of-memory errors. Undersized setups fail at the worst moments. Usually mid-demo. Ask me how I know. Actually, do not.
So: a 70B-class model is a single-high-end-GPU project at 4-bit, a multi-GPU project at higher precision, and a non-project on a laptop. Size honestly. Round up. Running out mid-inference is the most annoying way to learn any of this.
An LLM hardware requirement calculator tool makes it painless: your card, your model, what fits. And once it is running, a private AI deployment SaaS dashboard keeps the fleet visible, which models are loaded, where memory is going, what is idle. No more SSH guesswork.
Our quantized LLM VRAM requirements chart and LLM parameter count comparison table map this across 200 open models, so you match model to hardware before the download. Hardware Tetris not your thing? I do private AI setup for businesses at privateaiagent.fyi. Your data never leaves.