Local LLM hardware requirements: RAM, models and quantisation
A sober sizing guide for running 8B, 14B, 32B and 70B-class models on local hardware.
Model size determines whether a workload fits; memory bandwidth and acceleration largely determine whether it feels useful.
- 01Parameter count alone is not a speed estimate.
- 02Leave memory headroom for context, runtime, retrieval and the operating system.
- 03Quantisation trades memory and speed against model quality; test the actual workload.
Weights are only the start of the memory budget.
A rough four-bit estimate is about half a byte per parameter before format overhead. An 8-billion-parameter model therefore starts near 4 GB of weights, while a 70-billion-parameter model starts near 35 GB. Real deployments need more for metadata, runtime buffers, the context cache, retrieval services and the operating system.
Context length matters because the key-value cache grows as conversations and source material become longer. A model that starts comfortably can still run out of memory when a user attaches a large file set or requests an oversized context window.
| Model class | Practical system memory | Typical role |
|---|---|---|
| 7B–9B, 4-bit | 8–16 GB | Light drafting, extraction and compact assistants. |
| 12B–14B, 4-bit | 16–24 GB | Balanced office work and multilingual document tasks. |
| 27B–32B, 4-bit | 32–48 GB | Stronger drafting, coding and complex retrieval synthesis. |
| 70B–72B, 4-bit | 64 GB or more | Higher-quality local reasoning where latency is acceptable. |
Four-bit is a family of choices, not one setting.
Quantisation stores model weights with lower precision. This reduces memory use and often makes local inference practical, but different methods preserve quality differently. A label such as ‘4-bit’ does not identify the quantiser, calibration or runtime format.
For document work, the right test is not whether a compressed model completes a generic benchmark. It is whether it preserves names, numbers, citations, instructions and German-language nuance across representative files.
- —Keep the exact model revision and quantisation filename in the handover record.
- —Test at the intended context length, not only with one short prompt.
- —Recheck quality after any model or runtime update.
Tokens per second need context.
Prompt processing and answer generation are different phases. A system may ingest a long document slowly but generate the answer quickly, or the reverse. Published figures should therefore separate prompt-processing speed, generation speed, time to first token and the context length used.
CPU core count is a weak standalone predictor. Memory bandwidth, instruction support, GPU or integrated acceleration, thermal limits and runtime implementation often matter more. The same model file can behave very differently across appliances with the same nominal RAM.
Size for the daily model, not the largest model that can boot.
A system is useful when it has enough headroom for the model, the intended context and surrounding services without swapping or constant manual intervention. Running a larger model at unusable latency is not an upgrade.
SelbsAI therefore treats model class as an appliance-sizing decision. The selected Core or Pro system determines the attainable performance envelope.
Sources et méthode
Sources primaires et techniques consultées.
- 01llama.cpp repository and runtime documentationggerganov / GitHub · accessed 2026-08-09
- 02GGUF specificationggml-org / GitHub · accessed 2026-08-09
- 03Transformers quantization overviewHugging Face · accessed 2026-08-09
ALB Digital Dienstleistungen