Glyd
GitHubGet started

Report · Sep 26, 2026 · Glyd

Qwen3-32B on one 48 GB GPU, bit for bit

Glyd holds Qwen3-32B's weights in 44.5 GB instead of 65.5, so it runs on one 48 GB RTX A6000 instead of two, every weight exact, faster to 32 sequences.

Qwen3-32B’s weights take 65.5 GB in bf16, more than one 48 GB GPU holds. Glyd stores the same weights in 44.5 GB, so the model runs on one RTX A6000 instead of two. Every weight decodes to exactly the bf16 value it was. On one GPU, Glyd generates more tokens a second than bf16 does on two up to 32 sequences a step, and fewer at 64.

What nvidia-smi showed

nvidia-smi with the model loaded: bf16 spills onto a second GPU, and Glyd runs on one with 5,802 of an RTX A6000’s 49,140 MiB to spare.1

In use, MiBbf16Glyd
GPU 044,55443,338
GPU 118,514—
GPU memory in use with Qwen3-32B loaded, as nvidia-smi reports it. The black line is one RTX A6000's 49,140 MiB. Lambda Cloud, 4× RTX A6000 (48 GB each), Sep 26, 2026.

The results

bf16 and Glyd side by side on the same machine; Qwen2.5-72B-Instruct went the same way, four 48 GB GPUs in bf16 and three with Glyd.

bf16GlydChange, %
Qwen3-32B
Weights, GB65.544.5−32
48 GB GPUs it takes21
Tokens/s, 1 sequence9.511.8+24
Tokens/s, 8 sequences74.494.9+28
Tokens/s, 32 sequences275.9289.9+5
Tokens/s, 64 sequences488.4400.5−18
MMLU, 1,000 questions, %78.578.0
Qwen2.5-72B-Instruct
Weights, GB145.497.8−33
48 GB GPUs it takes43
Tokens/s, 1 sequence4.56.4+42
Tokens/s, 8 sequences35.049.0+40
Tokens/s, 32 sequences134.6153.8+14
Tokens/s, 64 sequences254.3218.1−14
MMLU, 1,000 questions, %81.981.8
Tokens per second at 1, 8, 32 and 64 sequences: bf16 on two GPUs 9.5, 74.4, 275.9 and 488.4; Glyd on one GPU 11.8, 94.9, 289.9 and 400.5.
Sequences a stepbf16, two GPUsGlyd, one GPU
19.511.8
874.494.9
32275.9289.9
64488.4400.5
Tokens per second, Qwen3-32B on RTX A6000s: bf16 across two GPUs, Glyd on one. Higher is better; log scale. Lambda Cloud, Sep 26, 2026.

On an H100 SXM, where both fit on one 80 GB GPU, GPU time a token with q, k, v and gate, up merged as vLLM runs them:

Sequences a stepbf16, msGlyd, msChange, %
127.9024.50−12
829.7826.75−10
3231.4433.46+6
6433.5237.35+11

From 32 sequences Glyd takes longer than bf16 (the known gaps). The H100 log

Same weights, same model

Glyd changes how the weights are stored, not what they are: every weight decodes to exactly the bf16 value it was. The scores still move a hair, because the matrix products round a little differently from bf16’s, as any two GPU kernels do, so a few close answers can flip either way.2

Method

The options are explained in the GPU docs; the other GPUs’ numbers are on the benchmarks.

  • Speed: tokens per second at 1, 8, 32 and 64 sequences a step, with greedy decoding; on the H100, GPU time a token after the prompt, over 16 steps.
  • Like a server: q, k, v as one product and gate, up as another, for bf16 and Glyd alike, as vLLM runs them.
  • Memory: nvidia-smi during the runs, taken by e2e.py --smi.
  • Quality: MMLU, 1,000 questions, bf16 against Glyd in the same run.
  • Exactness: every matrix packed, unpacked and compared bit for bit.
  • Hardware: Lambda Cloud, 4× RTX A6000 (48 GB each). The driver, CUDA and PyTorch versions are in the log.

Reproduce it

The harness packs the weights, checks every one bit for bit, and runs bf16 and Glyd side by side (getting started sets it up):

git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Qwen3-32B --format auto --fused \
  --merge --baseline --batch 1,8,32,64

Raw logs

Every run’s log and the nvidia-smi output are in the repository: benchmarks/gpu/lambda-gpu_4x_a6000-20260926-084757.

Footnotes

  1. nvidia-smi counts in MiB: 49,140 MiB is 51.5 GB in decimal units, while Hugging Face lists the weights in decimal GB. ↩

  2. Floating-point addition is not associative, so two kernels that sum the same products in a different order can round the last bit differently. The weights themselves are identical. ↩