Llama 3.1 8B
meta-llama/Llama-3.1-8B-Instruct on Hugging Face
Weights in GPU memory
16.1 to 10.8 GB
33% less, every weight restored bit for bitSmallest single GPU
24 GB to 16 GB
Weights, an 8K-token KV cache and 1.5 GB for the runtimeH100 80 GB GPUs it takes
1 to 1
Weights, an 8K-token KV cache and 1.5 GB a GPU for the runtimeMMLU, 300 questions
71.7% to 72.0%
Same weights; the products sum in another orderWhich single GPU it fits
Worked out from the measured weights.
| Memory | GPUs | bf16 | Glyd |
|---|---|---|---|
| 16 GB | RTX 4080, RTX 5080Room for about 36K tokens of context with Glyd | No | Fits |
| 24 GB | RTX 4090, RTX 3090, A10Room for the KV cache: about 62K tokens in bf16, 102K with Glyd | Fits | Fits |
| 32 GB | RTX 5090 | Fits | Fits |
| 48 GB | RTX A6000, L40S, RTX 6000 Ada | Fits | Fits |
| 80 GB | H100, A100 80 GB | Fits | Fits |
| 96 GB | RTX PRO 6000, GH200 | Fits | Fits |
| 141 GB | H200 | Fits | Fits |
Every matrix, bit for bit
Every Linear layer's matrix packed and unpacked; H100 SXM, Sep 27, 2026.
| Matrices in bf16 | 14 GB |
|---|---|
| With Glyd, tiered layout | 9.4 GB |
| Change | −32.8% |
The 12-bit layout takes 24.7 to 24.8% off each matrix and decodes with less work. The ten-model report
Same weights, same model
Every weight decodes to exactly the bf16 value it was. The matrix products add their terms in a different order than cuBLAS does, as any two GPU kernels do, so a few close answers can flip either way.
| Benchmark | bf16 | Glyd |
|---|---|---|
| MMLU, 300 questions | 71.7% | 72.0% |
| Perplexity, enwik8 | 19.5915 | 19.5909 |
H100 SXM, Sep 27, 2026. Lower perplexity is better.
Run it
git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Llama-3.1-8B-Instruct --format auto --fused \
--merge --baseline --batch 1,8,32Today Glyd runs from its PyTorch harness; vLLM and SGLang integration is not built yet.
Also measured
Twenty more open models measured, with quality wherever bf16 fits one GPU.