Qwen3 32B
Qwen/Qwen3-32B on Hugging Face
Memory on RTX A6000s, 48 GB each
nvidia-smi with the model loaded, from the run's log. 49,140 MiB a GPU.
- bf16, GPU 044,554 MiB
- bf16, GPU 118,514 MiB
- Glyd, GPU 043,338 MiB
In bf16 the model spills onto a second GPU. With Glyd it runs on one, with 5,802 MiB to spare.
Which single GPU it fits
48 GB measured; the others worked out from the measured weights.
| Memory | GPUs | bf16 | Glyd |
|---|---|---|---|
| 16 GB | RTX 4080, RTX 5080 | No | No |
| 24 GB | RTX 4090, RTX 3090, A10 | No | No |
| 32 GB | RTX 5090 | No | No |
| 48 GB | RTX A6000, L40S, RTX 6000 AdaMeasured with nvidia-smi | No | Fits |
| 80 GB | H100, A100 80 GBRoom for the KV cache: about 70K tokens in bf16, 150K with Glyd | Fits | Fits |
| 96 GB | RTX PRO 6000, GH200 | Fits | Fits |
| 141 GB | H200 | Fits | Fits |
Tokens per second, RTX A6000
Higher is better. bf16 across two GPUs, Glyd on one.
- 1 sequence+24% tokens/s
- 8 sequences+28% tokens/s
- 32 sequences+5% tokens/s
GPU time per token, H100 SXM
Lower is better. q, k, v and gate, up merged, as vLLM runs them.
- 1 sequence12% less time
- 8 sequences10% less time
- 32 sequences6% more time
- 64 sequences11% more time
Same weights, same model
Every weight decodes to exactly the bf16 value it was. The matrix products add their terms in a different order than cuBLAS does, as any two GPU kernels do, so a few close answers can flip either way.
| Benchmark | bf16 | Glyd |
|---|---|---|
| MMLU, 1,000 questions | 78.5% | 78.0% |
The same effect on other models goes both ways: Llama 3.1 8B 71.7% → 72.0%, Qwen3 8B 74.0% → 74.3%.
Run it
git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Qwen3-32B --format auto --fused \
--merge --baseline --batch 1,8,32Today Glyd runs from its PyTorch harness; vLLM and SGLang integration is not built yet.
Also measured
Twenty more open models measured, with quality wherever bf16 fits one GPU.