If you’ve ever tried to self-host a model and been hit with “CUDA out of memory” or seen someone mention INT4 quantization without understanding what was sacrificed, this is the breakdown you were missing.
Every term maps back to one question: can I actually run this thing on my hardware? Here’s the answer — visualized.
Parameters — The Size Question
Parameters are the core weights of a model. The “8B” in Llama-3 8B means approximately 8 billion trainable numbers — each storing learned knowledge from training. But parameter count alone doesn’t tell you what it takes to run. You also need to know the precision those parameters are stored at.
An 8B model is a spectrum of sizes depending on precision, not one fixed number. FP16 doubles that footprint versus INT4 — before we even discuss inference overhead.
Quantization — Trading Precision for Space and Speed
Quantization reduces the bit depth each parameter occupies: from 16-bit floats (FP16) down to 8-bit (FP8) or 4-bit integers (INT4). The goal is keeping the model “close enough” while making it dramatically smaller.
Going from FP16 to INT4 cuts VRAM usage by 75% — an 8B model drops from ~16 GB down to ~3.2 GB. The quality difference between FP8 and INT4 is often negligible for conversational use, making INT4 the practical sweet spot for most consumer GPUs.
KV Cache — Why Long Context Eats Memory
During inference, models store key-value pairs for every token they’ve processed so far. This cache lets subsequent tokens skip redundant computation instead of re-scanning earlier context. The cost is linear — each new token adds to memory with no way to reclaim it until the sequence ends.
For an 8B model in INT4, ~1 GB of extra VRAM per 10k tokens. A 128-token prompt is cheap; a 100k-token paste is expensive. The formula is:
Tokens × Layers × Heads × Dimension = KV Cache Size
Which means longer prompts eat VRAM even after you’ve already quantized the model down aggressively.
Dense vs Mixture-of-Experts — Two Ways to Use Parameters
Not all parameters get loaded for every token. Dense models activate everything uniformly, computing across all layers regardless of input. MoE (Mixture-of-Experts) routes each token through a small subset of specialized sub-networks called “experts.”
Gemma 7B (dense) computes all 7 billion parameters on every token. An MoE model with the same total parameter count might only compute ~1.8B per token — trading peak quality for dramatically lower per-token latency and memory bandwidth.
Tradeoff Summary
The decision comes down to what you’re optimizing for:
| Format | VRAM (8B) | Quality | Inference Speed | Best For |
|---|---|---|---|---|
| FP16 | 16 GB | Best | Slow | Server GPUs, high accuracy |
| FP8 | 8 GB | Good | Fast | Mid-range consumer cards |
| INT4 | ~3.2 GB | Acceptable | Fastest | Local inference, low VRAM |
For local deployment on a consumer GPU (say, an RTX 4060 with 8GB VRAM), INT4 is the only format that fits. The quality loss from FP16 is real but often imperceptible in everyday use — code generation and reasoning tasks stay sharp well into the INT4 range.
Bottom Line
Parameters tell you total model size, quantization tells you what it actually takes to run, KV cache tells you whether long prompts will kill your VRAM, and architecture tells you if every parameter matters on every token. Keep these four levers in mind and you’ll stop guessing when a new model drops.