Measured on real GPUs, bf16 and Glyd in the same run, the 2026 models first and each earlier release's tables as measured. Every number links to the run's raw log, and every run can be repeated from the repository.
Updated Oct 5, 2026RTX 4090 · RTX 5090 · H100 SXM · H100 PCIe · A100 · A10 · A10G · RTX PRO 6000 · RTX 4080 SUPER
2026 models: the weights in vLLM
The weights as vLLM 0.30.0 gives them, Glyd against vLLM's own bf16 on the same GPU, a row a release run. Lower is smaller.
RTX 4090, Qwen3.5-9B: bf16 did not start on this GPU with vLLM's default settings (out of memory at a 4,096-token context); glyd run runs the model with a 189,440-token context.
H100 SXM, Qwen3.8-27B: The KV cache held 204,117 tokens against bf16's 122,538 (1.67 times).
Parts of these models are not compressed yet (the vision tower, the multi-token-prediction head and the linear-attention layers), and on the RTX 5090 and the H100 Glyd picks a larger layout by default because it is faster there: the known gaps. Qwen3.8-27B's serving run: below. The tables that follow are the earlier releases' measurements, on the models of 2025, as measured.
Model files, v0.29.0
The 21 files cut from 2026 models in the record (Qwen3.8, Gemma 4, GLM-5.3-Flash, MiniMax-M3 and Mistral Small 4; 3.72 GB of BF16, FP8, Q8_0 and Q4_K_M files), as one set, computed from the record's table, each file weighted by its size: how much smaller each program makes the set, every file read back bit for bit. Higher is smaller.
How fast it responds (v0.25.1, Qwen3-8B and Qwen3-32B)
Through generate(), greedy: Glyd's default against bf16 compiled the same way, each in a process of its own. Tokens a second after a 128-token prompt, 256 new; the first token of a 2,048-token prompt; a chat (a 200-token prompt, a 300-token reply) and a long document (2,000 and 200), each to its last token. Over 1 is faster for tokens a second, under 1 sooner for the times.
The A100's and GH200's at v0.25.1, before v0.26.0's faster long prompts there. Every time and rate
Serving with vLLM
vllm bench serve, bf16 against --quantization glyd at the same memory setting: servers warm, 1,024 tokens in and 256 out, vLLM 0.30.0; one GPU each, and Qwen3-30B-A3B is a mixture of experts. Low load: 1 request a second, 0.25 on the L4; saturated: every request at once. Higher is better for the ratios, lower for the times.
Qwen2.5-72B on two H100s: the KV cache holds 111,328 tokens with Glyd and 8,800 with bf16, room for about 27 conversations of 4,096 tokens at once against 2. The report
The memory the packed weights save goes to vLLM's KV cache, so more requests run at once. These rows are v0.26.0's and v0.27.0's plugin, before the embedding and the output layer were packed too (the L4 against v0.28.0); Qwen3-30B-A3B on the H100 SXM, 0.84 times here, is served at 1.07 times by v0.28.0 (mixtures of experts). The A100, GH200 and H100 rows were measured again with each pass's prompts new to the server; the L4's and the A10's saturated passes served none from vLLM's prefix cache. 2× RTX A6000: being re-measured after a benchmark fix. Serving with vLLM, its options and exact mode. Every rate and the checks, as of v0.26.0
The whole model smaller, on an L4
From v0.28.0 vLLM packs the embedding and the output layer too. v0.27.0 from PyPI against v0.28.0 on one L4, vLLM 0.30.0 in its default mode (compiled, CUDA graphs), alternated in one session. Weights and KV cache as vLLM's log gives them with its own KV cache (--max-model-len 4096, 0.9 of the GPU's memory); the first start is on an empty compile cache. Higher is better for the ratios.
One user: one request of a 1,024-token prompt decoded for 256 tokens, the median of three rounds with the two alternating. Serving is vllm bench serve as users run it, the KV cache at its default (auto: lossless on an L4), a seed of its own for each pass, 64 requests at 1 a second and then 256 at once, the prefix cache's hit rate in the servers' logs at most 0.4%. The faster steps of the smallest layout, which an L4 takes by default, give 1.042 times v0.27.0's tokens a second on Qwen3-8B with the embedding and the output layer left as vLLM runs them (22.07 against 21.19); with them packed too, 1.065. The output layer's product is within the same tolerance as every other product; exact is unchanged. What is packed and what stays
Mixtures of experts in vLLM
vllm bench serve, 1,024 tokens in and 256 out, vLLM 0.30.0: 64 requests at 1 a second, then 256 at once, 0.9 of the GPU's memory, every pass with prompts of its own. The baseline is vLLM's own bf16 path with its default, untuned MoE settings (its log: Using default MoE config. Performance might be sub-optimal!); a tuned bf16 would be faster than that baseline, which was not measured. Higher is better for the requests a second, lower for the times.
Glyd serves more requests a second because the weights leave room for more of them at once; each token then takes longer, and the first token comes sooner. Prompt steps are where Glyd is behind: on the H100, with the lossless cache, 5.7 s of the saturated pass's 14.3 s were prompt steps, 22 microseconds a prompt token against bf16's 15. Qwen3-16B-A3B is Qwen3-30B-A3B with half its experts; the 30B does not fit this GPU in bf16. The attention products and the router stay bf16. Both runs are of the code as it stood on 2026-10-03. Mixtures of experts in the docs
Newer models in vLLM
vllm bench serve, 1,024 tokens in and 256 out, vLLM 0.30.0: 64 requests at 1 a second, then 256 at once, 0.9 of the GPU's memory, every pass with prompts of its own, against vLLM's own bf16. Higher is better for the requests a second, lower for the times.
Qwen3.8-27B, text only (the vision tower not loaded). Its attention layers keep vLLM's own KV cache: the lossless KV cache does not hold linear attention or a head size of 256 yet, so the room for more requests comes from the weights alone. bf16's continuations scored on Glyd: top-1 99.22%, as on bf16 started again. Glyd's saturated pass drew more power and ran at a lower GPU clock (1,852 MHz against 1,980). The code of v0.28.0rc1, 2026-10-03.
The lossless KV cache in vLLM
kv: lossless, vLLM's KV cache held in fewer bits with every value read back bit for bit, against vLLM's own cache: Qwen3-8B unless it says otherwise, bf16 weights, vLLM 0.30.0 in its default mode (compiled, CUDA graphs). Decode: 128 new tokens a request, at the same batch, the median of three rounds with the two alternating on the L4 and the A100, one run each on the H100 and the GH200. Higher is better for the ratios.
The first four rows are each decode run's own setting (0.96 of the GPU's memory on the L4, 0.9 on the A100, the H100 and the GH200). The other five: an L4, eager, --max-model-len 8704, 0.9, five models.
Decode tokens a second
GPU
Batch × tokens
vLLM's own
Lossless
×
L4
1 × 1,024
16.6
16.7
1.007
6 × 4,096
77.5
81.6
1.053
24 × 1,024
289.5
302.4
1.047
1 × 8,192
15.5
15.9
1.023
3 × 8,192
38.8
40.8
1.052
A100 SXM4 40 GB
1 × 1,024
75.3
76.7
1.018
1 × 8,192
71.4
73.7
1.032
H100 SXM
1 × 1,024
152.8
153.6
1.005
1 × 8,192
144.8
148.1
1.023
4 × 8,192
499.3
514.5
1.030
32 × 1,024
3,811.2
4,030.1
1.057
32 × 8,192
1,733.9
2,075.4
1.197
46 and 60 × 8,192
1,950.6
2,618.4
1.342
GH200
1 × 1,024
176.6
177.2
1.003
1 × 8,192
166.4
168.2
1.011
4 × 8,192
576.6
584.0
1.013
32 × 1,024
4,566.2
4,661.5
1.021
32 × 8,192
2,051.2
2,435.0
1.187
58 and 76 × 8,192
2,395.1
3,126.6
1.305
On the H100 and the GH200, the last row is each cache at the largest batch it holds: 46 requests of 8,192 tokens with vLLM's own, 60 with the lossless cache; the GH200's, 58 and 76.
With long prompts it serves more requests a second and its first token comes sooner; on the H100 each generated token takes longer, on the GH200 it takes less. An H200 is not measured; auto holds the cache on an A100, an L4, an H100 and a GH200 only. The option, its defaults and what is exact.
Requests a second
Lossless against vLLM's own, ×
1,024 tokens in, 256 out, saturated
vLLM's own
Lossless
×
First token
Each token
KV cache
L4bf16 weights, the lossless cache against vLLM's ownlog
0.76
0.93
1.23
0.81
1.06
1.29
L4Glyd's weights and the lossless cache against bf16 and vLLM's ownlog
0.76
1.20
1.59
0.58
1.55
2.64
A100 SXM4 40 GBbf16 weights, the lossless cache against vLLM's ownlog
5.92
7.62
1.29
0.81
0.93
1.30
H100 SXMbf16 weights, the lossless cache against vLLM's ownlog
20.58
21.68
1.05
1.02
0.92
1.31
GH200bf16 weights, the lossless cache against vLLM's ownlog
21.18
22.47
1.06
1.01
0.91
1.31
Every pass draws prompts of its own: the prefix cache's hit rate in the servers' logs was 0.0% on the L4 and the GH200, at most 0.9% on the A100 and 1.4% on the H100 (one prompt of 328 repeated). The second row is Glyd's weights and the lossless cache against bf16 and vLLM's own: each token takes longer there, the first token sooner.
A fraction of the layers, on an H100 SXM
Qwen3-32B on an 80 GB H100 SXM, bf16 against --quantization glyd with fraction, the share of the decoder layers packed (0 is bf16, 1 every layer). Every request sent at once: 192 prompts, 1,024 tokens in and 256 out, each server started cold with the same memory setting, --max-model-len 4096 --max-num-seqs 128, one run each. Glyd's layout is the 12-bit one. Higher is better for the ratios, lower for the times.
One GPU, one model and one load: the first token at low load was not measured here, nor a fraction on a GH200, where the same model served 0.89 times bf16's requests a second with every layer packed in the table above (another bench: warm servers, 128 prompts at once). bf16 ran at most 25 requests at a time with up to 175 waiting, every layer packed at most 81. The option.
What bf16, FP8 and Glyd change in a model's answers
Qwen3-4B-Instruct-2507 on an RTX 4080 SUPER, each mode against bf16's own answers: Glyd 0.22.0, and FP8 as Qwen releases it (Qwen/Qwen3-4B-Instruct-2507-FP8).
The weights are exact. Every weight Glyd holds decodes to its bf16 value, in both modes.
With exact=True the model is bf16's, bit for bit: the same perplexity, top tokens and greedy answers, and 0 MMLU answers changed.
By default, about as much as bf16's own attention kernels: 13 MMLU answers changed, against 16 for bf16 with eager attention in place of SDPA. Glyd's products round a little differently from bf16's.
FP8 changes 3.5 times as many: 46 MMLU answers, and 14 of 50 greedy answers are bf16's.
Perplexity on WikiText-2 (64 windows of 1,024 tokens); 50 greedy answers of 64 tokens; MMLU on 1,000 questions, 0-shot. Weights: memory after loading, Glyd's with its scratch. The run
Against DFloat11 on a 16 GB GPU
Qwen3-8B on an RTX 4080 SUPER, where its 16.38 GB of bf16 does not fit; both hold the weights in about 11.2 GB. Higher is better.
Qwen3-8B, layer 18's matrices on an RTX 4080 SUPER, with the L2 cache flushed as ZipServ times itself. Lower is better; Glyd's time in bold where it is the fastest. Glyd's smallest layout at 1 to 32 tokens, its 12-bit layout at 64.
Through generate(), greedy: Glyd's default against bf16 compiled the same way, each in a process of its own.
Serving
vllm bench serve, bf16 against --quantization glyd at the same memory setting, servers warm (the fraction sweep's and the Qwen2.5-72B run's, cold).
Quality
Perplexity, the top token, greedy answers and MMLU, against bf16 in the same run.
Exactness
Every matrix packed, unpacked and compared bit for bit.
Hardware
An RTX 4080 SUPER; Lambda Cloud A10, A100, H100, GH200 and RTX A6000; AWS A10G, L4, L40S and RTX PRO 6000; Runpod RTX 4090, RTX 5090 and H100 SXM for the 2026 models. Versions in each log.
Releases
Each table names the release and the day it was measured; earlier releases' tables stay as measured, the 2026 models' first.
v0.29.0: 33% on the newest modelsQwen3.5-9B's weights are 28.2% under bf16's on an RTX 4090 and 21.3% on an RTX 5090; Qwen3.8-27B's 22.8% on an H100. Not compressed yet: vision towers, multi-token-prediction heads and the linear-attention layers. The RTX 5090, the A10, the A100 and the H100 take a larger layout by default because it is faster there; the smallest becomes the default where it is as fast.RTX 4090, RTX 5090H100 SXM
Speed at full load on an H100Qwen3.8-27B, 256 requests at once: 1.07 times bf16's requests a second and the first token after 0.84 times its time, but each token took 1.43 times as long (66.2 ms against 46.4). At 1 request a second the first token came after 227 ms against 199.H100 SXMThe run
The lossless KV cache and the newest attentionThe lossless KV cache does not hold linear attention or a head size of 256 yet, so Qwen3.5 and Qwen3.8 keep vLLM's own KV cache, and their room for more requests comes from the weights alone.H100 SXM
Glyd's own engineIt runs Gemma 4 31B and 12B (v0.31.0) on Linux with an RTX 30 series card or newer: text only, one request at a time. Gemma 4 26B-A4B, E4B and E2B are not supported yet.
FP8 and 4-bit modelsFP8 weights have 17 to 18% to take, measured on three (v0.21.0), but lossless FP8 checkpoints (GLM-5.3, DeepSeek-V4, Mistral Small 4) are not shipped yet: Glyd keeps their weights as they are. 4-bit weights have little left to take.The roadmap
vLLM on a GH200, at full load (v0.27.0)Saturated, Glyd served 0.93 and 0.89 times bf16's requests a second on a GH200, Qwen3-8B and Qwen3-32B. On an 80 GB H100 SXM, Qwen3-32B with every layer packed served 1.56 times bf16's (another bench, where bf16 was short of KV cache); Qwen3-8B, which leaves bf16 room to spare, was not served at full load there.GH200H100 SXM, Qwen3-32B
Mixtures of experts in vLLM, prompts (v0.28.0)At full load a mixture of experts serves faster than bf16 since v0.28.0: 1.07 times its requests a second on an H100 SXM and 1.50 times with the lossless KV cache on an A100 40 GB, against vLLM's own bf16 path with its default MoE settings (a tuned bf16 would be faster; not measured). Its prompt steps are slower: 22 microseconds a prompt token against bf16's 15 on the H100 and 72 against 52 on the A100, and the experts' layer takes 1.38 to 2.01 times bf16's time at 1,152 to 8,192 tokens a step on the H100. At 1 request a second the first token on the A100 comes 8% later (110 ms against 102). The 2× RTX A6000 row is being re-measured after a benchmark fix.H100 SXMA100 40 GB
Qwen3-8B in vLLM on an A100, at full load (v0.27.0)Saturated, Glyd served 0.98 times bf16's requests a second (5.73 against 5.85) with 1.14 times the KV cache, and each token came 17% later; with Qwen3-14B, where bf16 is short of KV cache, 1.28 times.A100 40 GB
The lossless KV cache on an H100, with long prompts (v0.27.0)With 8,192 tokens in and 256 out, saturated, on an H100 SXM it served 1.05 times vLLM's requests a second and its first token came after 0.89 times the time, but each generated token took 1.12 times as long (49.2 ms against 43.9). On a GH200 the same run served 1.12 times the requests a second, with the first token after 0.95 times the time and each token 0.97 times as long. An H200, an L40S, an RTX 40 and an A10 are not measured: the default leaves vLLM's own cache there.H100 SXMGH200
Prompts of rare tokens in the lossless KV cache (v0.27.0)40 prompts of a model's rare tokens, 1,400 tokens each, at once with 8 ordinary prompts: all 40 served on every one of five models, and the flood ran at 0.88 times (Llama-3.1-8B) and 0.77 times (Mistral-7B-v0.3) vLLM's prompt tokens a second, and at 1.00 to 1.01 times on the Qwen models, which have no such token.L4
The first tokenAt low load the first token comes 6 to 27% later than bf16's in vLLM (27% for Qwen3-32B on a GH200, v0.27.0), and at most 31% through generate() (a 512-token prompt on an A100 SXM, v0.25.1).vLLM, GH200vLLM, A100generate(), A100
generate() at 8 sequences on an A100.94 times bf16 compiled's tokens a second, both eager there (v0.25.1).A10
Long prompts on datacenter GPUsStill slower than bf16: Qwen3-8B's pass over 1,024 tokens takes 101.2 ms against bf16's 93.0 on an A100 SXM (v0.26.0) and 45.0 against 36.9 on an H100 SXM (v0.24.0), and Qwen3-14B's over 2,048 tokens 134.4 against 119.2 on an H100 SXM (v0.26.0); work is under way.A100 SXMH100 SXMH100 SXM, 14B
Prompts on the L4 and L40SLong prompts are close to bf16's since v0.25.1: Qwen3-8B at 8,192 tokens +4.7% on an L4 and +3.8% on an L40S. Shorter ones in the smallest layout are still slower, +25.5% and +30.3% at 1,024 tokens; layout="mma12" is faster there.L4 and L40S
Many sequences a stepFrom 64 or 128 sequences a step, 2 to 10% more GPU time than bf16 on the A100, H100 PCIe, RTX PRO 6000 and RTX 4080 SUPER (v0.22.0).A100H100 PCIeRTX PRO 6000RTX 4080 SUPER
Exact mode and the longest prompts, on an A10Since v0.25.0 exact mode and the longest prompts take 1.0 to 1.6% longer on an A10; on Hopper v0.25.1 made them 2.5 to 3.6% faster than before, measured on a GH200 (the H100 SXM not run again).A10GH200
BlackwellMeasured on two Blackwell GPUs, the RTX PRO 6000 (v0.22.0) and the RTX 5090 (v0.29.0); not yet on the B200.Its numbers
