Release · Oct 4, 2026 · Glyd
Glyd v0.28.0: the whole model smaller
Glyd v0.28.0 packs the embedding and the output layer under vLLM: Qwen3-8B's weights are 32% under bf16's. Mixtures of experts serve faster than bf16.
Glyd v0.28.0 packs the whole model under vLLM. The embedding and the output layer, which stayed bf16 in v0.26.0 and v0.27.0, are compressed too, so Qwen3-8B’s weights on an L4 are 10.38 GiB against bf16’s 15.27, 32.0% smaller, and the memory this frees becomes KV cache. Mixtures of experts now serve faster than bf16 on an H100 and an A100, and the smallest layout runs one user’s tokens faster. Saves have a new format, and earlier saves still load.
The whole model
On an L4, v0.27.0 from PyPI against v0.28.0, the same vLLM 0.30.0, alternated in one session. vLLM’s own KV cache, --max-model-len 4096, 0.9 of the GPU’s memory, as vLLM’s log gives the weights and the cache; the first start is on an empty compile cache:
| Model | Weights, GiB | KV cache, tokens | First start, tokens | |||||
|---|---|---|---|---|---|---|---|---|
| bf16 | v0.27.0 | v0.28.0 | bf16 | v0.27.0 | v0.28.0 | v0.27.0 | v0.28.0 | |
| Qwen3-8B log | 15.27 | 11.38 | 10.38 | 27,024 | 54,464 | 60,736 | 38,912 | 60,736 |
| Llama-3.1-8B | 15.00 | 11.05 | 10.16 | 32,592 | 64,368 | 70,112 | 50,032 | 70,112 |
| Qwen3-4B | 7.56 | 5.48 | 5.16 | 83,408 | 97,488 | 99,312 | 88,048 | 99,120 |
Every row of the embedding comes back bit for bit, the output layer’s product is within the same tolerance as every other product, and a model whose config ties the two keeps one packed copy. exact is unchanged: vLLM’s bf16 logits, bit for bit. A first start, on an empty compile cache, no longer logs a much smaller KV cache than a later one (38,912 tokens against 54,464 for Qwen3-8B with v0.27.0; v0.28.0’s 60,736 on both).
One user, Qwen3-8B, 1,024 tokens in and 256 out, three rounds alternated: 22.54 tokens a second against v0.27.0’s 21.16, 1.065 times. The faster steps of the smallest layout alone, with the embedding and the output layer left as vLLM runs them, give 1.042 times (22.07 against 21.19); the layout is the L4’s default, and on the GPUs that take the mma12 layout by default nothing changes.
With vllm bench serve as users run it, 1,024 tokens in and 256 out, the KV cache at its default (auto, lossless on an L4), saturated, Qwen3-8B served 1.33 and 1.31 requests a second against v0.27.0’s 1.21 and 1.20 in two rounds, with a KV cache of 79,600 tokens against 71,248. Every model and rate: the benchmarks.
They stay as vLLM runs them, with a line in the log that says why, under a fraction below 1, speculative decoding, exact (the output layer alone), an output layer that is not bf16, an embedding class that is not a plain lookup and a hidden size that is not a multiple of 64. Tensor parallel 2 was checked on two L4s (the log).
Mixtures of experts
Against vLLM’s own bf16 path with its default MoE settings (its log says Using default MoE config. Performance might be sub-optimal!; a tuned bf16 would be faster than that baseline, which was not measured), vllm bench serve, 1,024 tokens in and 256 out, saturated, vLLM 0.30.0:
| GPU, model | Mode | Requests a second | First token, ms | Each token, ms | |||
|---|---|---|---|---|---|---|---|
| Rate | × | Time | × | Time | × | ||
| H100 SXM, Qwen3-30B-A3B log | bf16 | 12.32 | 1.00 | 6,920 | 1.00 | 27.2 | 1.00 |
| Glyd | 13.13 | 1.07 | 3,853 | 0.56 | 45.9 | 1.69 | |
| Glyd, lossless KV cache | 17.10 | 1.39 | 3,437 | 0.50 | 43.7 | 1.61 | |
| A100 SXM4 40 GB, Qwen3-16B-A3B log | bf16 | 4.12 | 1.00 | 27,100 | 1.00 | 30.9 | 1.00 |
| Glyd, lossless KV cache | 6.18 | 1.50 | 14,233 | 0.53 | 62.9 | 2.04 | |
Qwen3-16B-A3B is Qwen3-30B-A3B with half its experts; the 30B does not fit an A100 40 GB in bf16. The gain is room: the H100’s weights take 44.65 GiB against bf16’s 56.88 and its KV cache holds 251,696 tokens against 118,720; the A100’s weights take 23.97 GiB against 29.87 and its KV cache 144,896 tokens against 40,288. Each token takes longer at saturation and the first token comes sooner. At 1 request a second both served the same requests a second as bf16: on the H100 each token took 0.95 times as long and the first token 0.90 times; on the A100 each token 0.88 times and the first token 1.08 times as long (110 ms against 102).
Prompt steps are where Glyd is behind: on the H100, with the lossless KV cache, 5.7 s of the saturated pass’s 14.3 s were prompt steps, 22 microseconds a prompt token against bf16’s 15. The attention products and the router stay bf16 in these models. Both runs are of the code as it stood on 2026-10-03, with no setting changed by hand (the known gaps, the docs).
A newer model
Qwen3.8-27B on an H100 SXM, text only, under vLLM 0.30.0: the weights take 38.77 GiB against bf16’s 50.22 (22.8% smaller, the 12-bit layout this GPU takes by default) and vLLM’s KV cache holds 204,117 tokens against 122,538. With 256 requests at once that is 1.07 times bf16’s requests a second, the first token after 0.84 times the time and each token 1.43 times as long; at 1 request a second each token takes 0.92 times as long. Its linear-attention layers and head size of 256 keep vLLM’s own KV cache for now. The run’s log; the table.
Saves, and the C API
glyd pack and save_pretrained write the smallest layout as format glyd-v4 (--layout mma12 stays glyd-v3). Every earlier save loads, a glyd-v1 or glyd-v2 save being packed again as it loads; v0.27.0 and earlier refuse a glyd-v4 save, by its format. The C API is version 15, with the same 8 functions (the CUDA library). The mma12 layout stays the default on the A10, A100, H100 and GH200, and the KV cache’s defaults are v0.27.0’s.
Where Glyd is still slower than bf16: the known gaps. Every change, with its numbers: the changelog and the release notes.