Qwen3 4B 2507
Qwen/Qwen3-4B-Instruct-2507 on Hugging Face
Weights in GPU memory
8.0 to 5.5 GB
32% less, every weight restored bit for bitSmallest single GPU
16 GB to 16 GB
Weights, an 8K-token KV cache and 1.5 GB for the runtimeH100 80 GB GPUs it takes
1 to 1
Weights, an 8K-token KV cache and 1.5 GB a GPU for the runtimeMMLU, 300 questions
71.0% to 71.0%
Same weights; the products sum in another orderWhich single GPU it fits
Worked out from the measured weights.
| Memory | GPUs | bf16 | Glyd |
|---|---|---|---|
| 16 GB | RTX 4080, RTX 5080Room for the KV cache: about 51K tokens in bf16, 69K with Glyd | Fits | Fits |
| 24 GB | RTX 4090, RTX 3090, A10 | Fits | Fits |
| 32 GB | RTX 5090 | Fits | Fits |
| 48 GB | RTX A6000, L40S, RTX 6000 Ada | Fits | Fits |
| 80 GB | H100, A100 80 GB | Fits | Fits |
| 96 GB | RTX PRO 6000, GH200 | Fits | Fits |
| 141 GB | H200 | Fits | Fits |
Every matrix, bit for bit
Every Linear layer's matrix packed and unpacked; A10, Sep 27, 2026.
| Matrices in bf16 | 7.3 GB |
|---|---|
| With Glyd, tiered layout | 4.9 GB |
| Change | −32.2% |
The 12-bit layout takes 24.7 to 24.8% off each matrix and decodes with less work. The ten-model report
Same weights, same model
Every weight decodes to exactly the bf16 value it was. The matrix products add their terms in a different order than cuBLAS does, as any two GPU kernels do, so a few close answers can flip either way.
| Benchmark | bf16 | Glyd |
|---|---|---|
| MMLU, 300 questions | 71.0% | 71.0% |
| Perplexity, enwik8 | 22.4636 | 22.4672 |
H100 SXM, Sep 27, 2026. Lower perplexity is better.
Run it
git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Qwen3-4B-Instruct-2507 --format auto --fused \
--merge --baseline --batch 1,8,32Today Glyd runs from its PyTorch harness; vLLM and SGLang integration is not built yet.
Also measured
Twenty more open models measured, with quality wherever bf16 fits one GPU.