Report · · Glyd
Qwen3-32B on one 48 GB GPU, bit for bit
Glyd holds Qwen3-32B's weights in 44.5 GB instead of 65.5, so it runs on one 48 GB RTX A6000 instead of two, every weight exact and faster.
Qwen3-32B’s weights take 65.5 GB in bf16, more than one 48 GB GPU holds. Glyd stores the same weights in 44.5 GB and decodes them inside the matrix multiply, so the model runs on one RTX A6000 instead of two. Every weight decodes to exactly the bf16 value it was, and on one GPU Glyd generates more tokens a second than bf16 does on two.
What nvidia-smi showed
The runs took nvidia-smi with the model loaded (e2e.py --smi). In bf16 the model spills onto a second GPU: 44,554 MiB on the first and 18,514 MiB on the second. With Glyd it runs on one, at 43,338 of the 49,140 MiB an RTX A6000 has, which leaves 5,802 MiB to spare.1
The results
The same runs, bf16 and Glyd side by side on the same machine. Qwen2.5-72B went the same way: four 48 GB GPUs in bf16, three with Glyd.
| The same runs | bf16 | Glyd |
|---|---|---|
| Qwen3-32B: weights | 65.5 GB | 44.5 GB |
| Qwen3-32B: 48 GB GPUs it takes | 2 | 1 |
| Qwen3-32B: tokens/s at 1 / 8 / 32 sequences | 9.5 / 74 / 276 | 11.8 / 95 / 290 |
| Qwen3-32B: MMLU, 1,000 questions | 78.5% | 78.0% |
| Qwen2.5-72B: weights | 145.4 GB | 97.8 GB |
| Qwen2.5-72B: 48 GB GPUs it takes | 4 | 3 |
| Qwen2.5-72B: tokens/s at 1 / 8 / 32 sequences | 4.5 / 35 / 135 | 6.4 / 49 / 154 |
| Qwen2.5-72B: MMLU, 1,000 questions | 81.9% | 81.8% |
| KV cache, Qwen2.5-7B, 16K tokens | 947 MB | 651 MB |
On one GPU, Glyd’s Qwen3-32B generates 24% more tokens a second than bf16 across two at one sequence, 28% more at 8 and 5% more at 32.
| Sequences a step | bf16, two GPUs | Glyd, one GPU |
|---|---|---|
| 1 | 9.5 | 11.8 |
| 8 | 74 | 95 |
| 32 | 276 | 290 |
On an H100 SXM, where both fit on one GPU, the model’s GPU time a token with q, k, v and gate, up merged as vLLM runs them was 12% lower with Glyd at one sequence and 10% lower at 8. At 32 and 64 sequences a step the decode work shows and Glyd took 6% and 11% more time.
Same weights, same model
Glyd changes how the weights are stored, not what they are: every weight decodes to exactly the bf16 value it was. The scores still move a hair, because the matrix products add their terms in a different order than cuBLAS does, as any two GPU kernels do, so a few close answers can flip either way.2 On 1,000 MMLU questions Qwen3-32B scored 78.5% in bf16 and 78.0% with Glyd; Qwen2.5-72B scored 81.9% and 81.8%.
Method
- Speed: tokens per second at 1, 8 and 32 sequences a step, with greedy decoding. On the H100, GPU time of a generated token apart from the prompt: 17 steps less 1, over 16.
- Like a server: q, k, v as one product and gate, up as another, for bf16 and Glyd alike, as vLLM runs them.
- Memory:
nvidia-smiduring the runs, taken bye2e.py --smi. - Quality: MMLU, 1,000 questions, bf16 against Glyd in the same run.
- Exactness: every matrix packed, unpacked and compared bit for bit.
- Hardware: Lambda Cloud, 4× RTX A6000 (48 GB each). The driver, CUDA and PyTorch versions are in the log.
Reproduce it
The harness packs the weights, checks every one bit for bit, and runs bf16 and Glyd side by side:
git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Qwen3-32B --format auto --fused \
--merge --baseline --batch 1,8,32Raw logs
Every run’s log and the nvidia-smi output are in the repository: benchmarks/gpu/lambda-gpu_4x_a6000-20260926-084757.
Footnotes
-
nvidia-smi counts in MiB: 49,140 MiB is 51.5 GB in decimal units, while Hugging Face lists the weights in decimal GB. ↩
-
Floating-point addition is not associative, so two kernels that sum the same products in a different order can round the last bit differently. The weights themselves are identical. ↩