What runs on a 24 GB GPU, exactly
RTX 4090, RTX 3090 and A10. Open models in their original precision, no quantization, with and without Glyd.
Runs only with Glyd
| Needs, GB | Room for context, K tokens | ||
|---|---|---|---|
| Model | bf16 | Glyd | Glyd |
 gemma-4-12B-itest. | 26.4 | 18.6 | 123 |
| gemma-3-12b-it | 26.9 | 18.8 | 119 |
 phi-4 | 32.6 | 23.0 | 22 |
 DeepSeek-R1-Distill-Qwen-14B | 32.8 | 23.3 | 20 |
Runs either way
| Needs, GB | Room for context, K tokens | |||
|---|---|---|---|---|
| Model | bf16 | Glyd | bf16 | Glyd |
 Qwen3.5-2Best. | 6.3 | 4.8 | 1,595 | 1,716 |
 SmolLM3-3B | 8.4 | 6.3 | 244 | 271 |
 Llama-3.2-3B-Instruct | 9.0 | 6.9 | 154 | 173 |
| NVIDIA-Nemotron-3-Nano-4B-BF16est. | 9.7 | 7.1 | 989 | 1,148 |
| Qwen3-4B-Instruct-2507 | 10.9 | 8.3 | 109 | 127 |
| Qwen3.5-4Best. | 11.2 | 8.1 | 452 | 546 |
| gemma-4-E2B-itest. | 11.9 | 8.6 | 1,939 | 2,407 |
 Mistral-7B-Instruct-v0.3 | 17.2 | 12.4 | 74 | 110 |
| gemma-4-E4B-itest. | 17.8 | 12.5 | 569 | 933 |
| Llama-3.1-8B-Instruct | 18.7 | 13.5 | 62 | 102 |
| Qwen3-8B | 19.2 | 14.0 | 53 | 88 |
 MiMo-V2.6-Distill-Qwen-9Best. | 20.7 | 14.5 | 163 | 351 |
| Qwen3.5-9B | 21.2 | 15.7 | 148 | 314 |
Too big for 24 GB
| Needs, GB | |||
|---|---|---|---|
| Model | bf16 | Glyd | Fits on |
| Mistral-Small-3.2-24B-Instruct-2506 | 51.0 | 35.1 | 48 GB |
| gemma-4-26B-A4B-it | 53.8 | 36.8 | 48 GB |
| gemma-3-27b-it | 57.6 | 39.6 | 48 GB |
| Qwen3.8-27B | 57.7 | 39.5 | 48 GB |
Speed on these GPUs
| GPU time a step, ms | |||
|---|---|---|---|
| Sequences a step | bf16 | Glyd | Change, % |
| 1 | 35.70 | 27.75 | −22 |
| 8 | 40.13 | 29.29 | −27 |
| 32 | 42.53 | 32.08 | −25 |
| 64 | 45.99 | 35.78 | −22 |
| GPU time a step, ms | |||
|---|---|---|---|
| Sequences a step | bf16 | Glyd | Change, % |
| 1 | 37.13 | 27.80 | −25 |
| 8 | 39.93 | 29.30 | −27 |
| 32 | 42.58 | 31.91 | −25 |
| 64 | 47.69 | 34.80 | −27 |
Questions about 24 GB GPUs
Which models run on a 24 GB GPU only with Glyd?
gemma-4-12B-it, gemma-3-12b-it, phi-4 and DeepSeek-R1-Distill-Qwen-14B. With an 8K-token context they need 26.4 to 32.8 GB in bf16, more than one 24 GB GPU holds, and 18.6 to 23.3 GB with Glyd, every weight exact.
Is it slower than bf16?
Usually not, at a few sequences a step: the table on this page gives v0.22.0's numbers. At many sequences, and for long prompts on some GPUs, Glyd still takes longer (the known gaps on the benchmarks page).
Has Glyd been measured on a 24 GB GPU with a 2026 model?
Yes: Qwen3.5-9B's weights took 12.68 GiB with Glyd on an RTX 4090 against bf16's 17.66 (vLLM 0.30.0, v0.29.0). bf16 did not start on this GPU with vLLM's default settings (out of memory at a 4,096-token context); glyd run runs the model with a 189,440-token context.
How is this different from a 4-bit GGUF?
A 4-bit model is smaller, 4 bits a weight and their scales, but its weights are rounded and its answers change. Glyd gives you the original model.