Qwen2.5 72B
Qwen/Qwen2.5-72B-Instruct on Hugging Face
Weights in GPU memory
145.4 to 97.8 GB
33% less, every weight restored bit for bit48 GB GPUs it takes
4 to 3
From the run, RTX A6000sTokens per second, 1 sequence
4.5 to 6.4
bf16 on four A6000s, Glyd on threeMMLU, 1,000 questions
81.9% to 81.8%
Same weights; the products sum in another orderWhich single GPU it fits
Worked out from the measured weights.
| Memory | GPUs | bf16 | Glyd |
|---|---|---|---|
| 16 GB | RTX 4080, RTX 5080 | No | No |
| 24 GB | RTX 4090, RTX 3090, A10 | No | No |
| 32 GB | RTX 5090 | No | No |
| 48 GB | RTX A6000, L40S, RTX 6000 Ada | No | No |
| 80 GB | H100, A100 80 GB | No | No |
| 96 GB | RTX PRO 6000, GH200Room for about 10K tokens of context with Glyd | No | Fits |
| 141 GB | H200Room for the KV cache: about 11K tokens in bf16, 157K with Glyd | Fits | Fits |
Tokens per second, RTX A6000
Higher is better. bf16 across four GPUs, Glyd on three.
bf16Glyd
- 1 sequence+42% tokens/s
- 8 sequences+40% tokens/s
- 32 sequences+14% tokens/s
Same weights, same model
Every weight decodes to exactly the bf16 value it was. The matrix products add their terms in a different order than cuBLAS does, as any two GPU kernels do, so a few close answers can flip either way.
| Benchmark | bf16 | Glyd |
|---|---|---|
| MMLU, 1,000 questions | 81.9% | 81.8% |
The same effect on other models goes both ways: Llama 3.1 8B 71.7% → 72.0%, Qwen3 8B 74.0% → 74.3%.
Run it
git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Qwen2.5-72B-Instruct --format auto --fused \
--merge --baseline --batch 1,8,32Today Glyd runs from its PyTorch harness; vLLM and SGLang integration is not built yet.
Also measured
Twenty more open models measured, with quality wherever bf16 fits one GPU.