What runs on a 96 GB GPU, exactly
RTX PRO 6000 and GH200. Open models in their original precision, no quantization, with and without Glyd.
Verified bit for bit, and timed against bf16 in the same run, on the RTX 4090, RTX 5090, H100 SXM, H100 PCIe, A100, A10, A10G, RTX PRO 6000 and RTX 4080 SUPER.
See the benchmarksRuns only with Glyd
| Needs, GB | Room for context, K tokens | ||
|---|---|---|---|
| Model | bf16 | Glyd | Glyd |
 Llama-3.3-70B-Instruct | 145.4 | 99.0 | 19 |
 Qwen2.5-72B-Instruct | 149.7 | 102.1 | 10 |
Runs either way
| Needs, GB | Room for context, K tokens | |||
|---|---|---|---|---|
| Model | bf16 | Glyd | bf16 | Glyd |
| Qwen3.5-2Best. | 6.3 | 4.8 | 7,852 | 7,973 |
 SmolLM3-3B | 8.4 | 6.3 | 1,287 | 1,314 |
| Llama-3.2-3B-Instruct | 9.0 | 6.9 | 825 | 843 |
| NVIDIA-Nemotron-3-Nano-4B-BF16est. | 9.7 | 7.1 | 5,681 | 5,841 |
| Qwen3-4B-Instruct-2507 | 10.9 | 8.3 | 631 | 648 |
| Qwen3.5-4Best. | 11.2 | 8.1 | 2,799 | 2,892 |
 gemma-4-E2B-itest. | 11.9 | 8.6 | 12,665 | 13,134 |
 Mistral-7B-Instruct-v0.3 | 17.2 | 12.4 | 660 | 696 |
| gemma-4-E4B-itest. | 17.8 | 12.5 | 5,932 | 6,296 |
| Llama-3.1-8B-Instruct | 18.7 | 13.5 | 648 | 688 |
| Qwen3-8B | 19.2 | 14.0 | 574 | 610 |
 MiMo-V2.6-Distill-Qwen-9Best. | 20.7 | 14.5 | 2,509 | 2,697 |
| Qwen3.5-9B | 21.2 | 15.7 | 2,494 | 2,660 |
| gemma-4-12B-itest. | 26.4 | 18.6 | 1,177 | 1,296 |
| gemma-3-12b-it | 26.9 | 18.8 | 1,170 | 1,292 |
 phi-4 | 32.6 | 23.0 | 350 | 397 |
 DeepSeek-R1-Distill-Qwen-14B | 32.8 | 23.3 | 364 | 412 |
| Mistral-Small-3.2-24B-Instruct-2506 | 51.0 | 35.1 | 324 | 420 |
| gemma-4-26B-A4B-it | 53.8 | 36.8 | 1,207 | 1,620 |
| gemma-3-27b-it | 57.6 | 39.6 | 564 | 783 |
| Qwen3.8-27B | 57.7 | 39.5 | 694 | 972 |
| Qwen3.5-27Best. | 57.7 | 39.5 | 694 | 971 |
| Qwen3.6-27Best. | 57.7 | 39.5 | 694 | 971 |
| Muse-Glimmer-30B | 61.4 | 41.8 | 3,116 | 4,583 |
| Qwen3-30B-A3B | 63.5 | 43.5 | 407 | 610 |
| gemma-4-31B-itest. | 66.3 | 45.9 | 235 | 360 |
| Qwen3-32B | 69.3 | 48.3 | 136 | 216 |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16est. | 67.5 | 46.0 | 5,729 | 9,232 |
| Qwen3.5-35B-A3Best. | 73.7 | 50.2 | 1,422 | 2,570 |
| Qwen3.6-35B-A3Best. | 73.7 | 50.2 | 1,422 | 2,570 |
Too big for 96 GB
| Needs, GB | |||
|---|---|---|---|
| Model | bf16 | Glyd | Fits on |
| Qwen3-Coder-Nextest. | 161.2 | 109.1 | 141 GB |
| Qwen3-Next-80B-A3B-Instruct | 164.5 | 112.1 | 141 GB |
| Llama-4-Scout-17B-16E-Instruct | 220.5 | 149.0 | 141 GB |
 GLM-4.5-Air | 224.1 | 151.2 | 2 H100s |
Speed on this GPU
| GPU time a step, ms | |||
|---|---|---|---|
| Sequences a step | bf16 | Glyd | Change, % |
| 1 | 13.40 | 11.29 | −16 |
| 8 | 14.03 | 12.18 | −13 |
| 32 | 15.31 | 13.86 | −9 |
| 64 | 16.32 | 16.77 | +3 |
| 128 | 18.44 | 18.84 | +2 |
Questions about 96 GB GPUs
Which models run on a 96 GB GPU only with Glyd?
Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct. With an 8K-token context they need 145.4 to 149.7 GB in bf16, more than one 96 GB GPU holds, and 99 to 102.1 GB with Glyd, every weight exact.
Is it slower than bf16?
Usually not, at a few sequences a step: the table on this page gives v0.22.0's numbers. At many sequences, and for long prompts on some GPUs, Glyd still takes longer (the known gaps on the benchmarks page).
How is this different from a 4-bit GGUF?
A 4-bit model is smaller, 4 bits a weight and their scales, but its weights are rounded and its answers change. Glyd gives you the original model.