Qwen2.5-72B on two H100s

Qwen2.5-72B-Instruct’s bf16 weights fill 67.80 GiB of each of two 80 GB H100s in vLLM, which leaves 1.34 GiB each for the KV cache: 8,800 tokens, room for about two conversations of 4,096 tokens at once. Glyd stores the same weights in fewer bits and rebuilds the exact values on the GPU. With it they fill 52.16 GiB on each GPU, which leaves 16.99 GiB each: a KV cache of 111,328 tokens, room for about 27 conversations at once.

What vLLM allocated

vLLM sizes its KV cache after the weights load, so the memory Glyd saves becomes room for requests. Each GPU’s KV cache with the server at --gpu-memory-utilization 0.9:

bf16

8,800 tokens

Glyd

111,328 tokens

050,000100,000

The results

vllm bench serve against each server, 1,024 tokens in and 256 out. The first token is vLLM’s time to first token, each next token its time per output token, and p99 the time 99% of requests came in under.¹

At 1 request a second, bf16’s server log shows up to 7 requests waiting and its first token took 2,802 ms on average; Glyd’s shows none waiting and its first token took 245 ms. Glyd served 1.19 times the requests a second, each next token in 0.95 times the time. With every request sent at once, bf16 ran up to 8 requests at a time and Glyd up to 98: Glyd served 4.07 times the requests a second (3.59 against 0.88), and its first token came in 0.18 times the time.²

Glyd is slower in two places. With every request sent at once each next token took 79.3 ms against 31.2 (2.54 times the time), with up to 98 requests running at once against 8. And starting the server cold took about 349 s against about 85 s.

Method

The options are explained in the vLLM docs; the other GPUs’ numbers are on the benchmarks.

• Serving: vllm serve over two GPUs with tensor parallelism, --max-model-len 4096 --gpu-memory-utilization 0.9 --max-num-seqs 128 for bf16 and Glyd alike, each server started cold, on an empty compile cache.
• Load: vllm bench serve, the random dataset (1,024 tokens in, 256 out, --ignore-eos): 64 requests at 1 request a second, then 192 at once. One run each.
• Prompts: each pass draws its prompts with a seed of its own (0, then 1), the same for bf16 and Glyd. vLLM’s prefix cache was on, and its hit rate is in the note under the results.
• Hardware and software: Lambda Cloud, 2× H100 SXM (80 GB each), driver 580.126.20; vLLM 0.30.0, PyTorch 2.13.0 with CUDA 13.0; Glyd v0.26.0 (10e8caea) with the bench scripts of 6a82f3bc, in the 12-bit layout the H100 takes (which layout on which GPU).

Reproduce it

The record’s README has every command, the job that ran them and the files. The commands that matter, for Glyd (leave --quantization glyd out for bf16):

pip install "glyd[vllm]==0.26.0"

vllm serve Qwen/Qwen2.5-72B-Instruct \
  --quantization glyd \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.9 \
  --tensor-parallel-size 2 \
  --max-num-seqs 128 \
  --compilation-config \
  '{"max_cudagraph_capture_size":128}'

vllm bench serve \
  --model Qwen/Qwen2.5-72B-Instruct \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 256 \
  --ignore-eos --seed 1 \
  --num-prompts 192 \
  --request-rate inf

That bench is the run with every request at once; the other is --num-prompts 64 --request-rate 1 --seed 0.

Raw logs

Both servers’ logs, every run’s result file, the nvidia-smi samples and the job are in the repository: benchmarks/gpu/h100x2-qwen2.5-72b-2026-10-03.

Footnotes

1. The ratios are Glyd’s over bf16’s, from the runs’ results before rounding, so 0.90 over 0.76 reads 1.19×. ↩

2. vLLM’s prefix cache was on, as it is when it serves: a prompt it already holds is not computed again. Each pass drew its prompts with a seed of its own (0, then 1). The servers’ logs show the cache’s hit rate: 0.0% for bf16 in every line, and for Glyd 0.0% through the first pass and 1.2% at most in the second, 0.4% over the whole run, which is about one prompt of the 256 sent. ↩