On the GPU
From Python, pip install "glyd[gpu]" and glyd.from_pretrained() load a model packed in the layout this page picks for your GPU (getting started); from C, C++ or Rust, the same kernels are the CUDA library. This page covers the layouts, and the harness the benchmarks were measured with.
The layouts
Glyd holds a model’s weights on the GPU in one of two layouts and picks one for your GPU. Bits a weight on Qwen2.5:
| Layout | Bits a weight |
|---|---|
mma, the smallest layout |
10.80 |
mma12, the 12-bit layout |
12.04 |
These are the layouts this site measures; e2e.py also takes the formats fast (its default) and huffman. The prebuilt library carries native code for Ampere, Ada, Hopper and Blackwell. Every product gives the same result every run. What each release changed, and by how much: the changelog.
Two layouts, and which GPU runs which
mma takes the most off. Which layout is the faster depends on the GPU and on the tokens a step. Qwen2.5-7B-Instruct, q, k, v and gate, up merged; run on Sep 26, 2026, on v0.20.0’s code:
| GPU time a token, ms | Sequences a step | |||
|---|---|---|---|---|
| 1 | 8 | 32 | 64 | |
| RTX 4080 SUPER 16 GB log | ||||
| bf16 | 21.97 | 22.85 | 26.19 | 27.67 |
Smallest, mma | 16.52 | 17.37 | 18.80 | 25.14 |
12-bit, mma12 | 17.48 | 18.25 | 19.67 | 21.66 |
| A10 24 GB log | ||||
| bf16 | 33.71 | 34.77 | 35.50 | 37.98 |
Smallest, mma | 24.33 | 25.61 | 35.99 | 49.66 |
12-bit, mma12 | 24.40 | 25.76 | 29.03 | 33.05 |
| A100 40 GB log | ||||
| bf16 | 14.18 | 14.87 | 16.18 | 17.80 |
Smallest, mma | 15.34 | 16.00 | 23.99 | 28.81 |
12-bit, mma12 | 11.84 | 13.77 | 17.16 | 19.95 |
| H100 SXM 80 GB log | ||||
| bf16 | 6.90 | 7.50 | 8.02 | 8.55 |
12-bit, mma12 | 6.55 | 7.24 | 8.69 | 9.99 |
Glyd’s time in bold where it is the fastest. On Ada the smallest layout, mma, is also the fastest to 32 sequences; on the others mma12 is, or within 1% of it. The table is run again at the next release’s sweep; each release since: tested GPUs. glyd_gpu.best_layout(), and e2e.py --format auto, take mma on Ada, for a mixture of experts on an A10 too, and wherever only it fits, mma12 elsewhere.
e2e.py, option by option
python e2e.py MODEL_DIR [options], from the repository’s gpu/ folder at v0.26.0, the last release that has it (the GPU code is compiled from v0.27.0):
| Option | What it does |
|---|---|
--format |
fast (the default), huffman, mma (the smallest layout), mma12 (the 12-bit layout) or auto (the one for this GPU) |
--fused |
run Glyd’s own products |
--exact |
bf16’s own products: the logits are bf16’s bit for bit (in place of --fused) |
--from-pretrained |
Glyd as glyd.from_pretrained loads it, the package’s own path (--format mma, mma12 or auto; --merge; --exact; one GPU), in place of the script’s packing |
--baseline |
run the model in bf16 too, in the same process, and compare |
--batch 1,8,32 |
generate for this many copies of the prompt at once, each size measured (default 1) |
--prompts |
a batch of different prompts, not copies of one: a mixture of experts routes each to its own experts |
--compile |
generate() compiled as transformers compiles it: a static cache, the forward under torch.compile with CUDA graphs |
--tokens 128 |
new tokens to generate (default 128) |
--merge |
the Linears that take the same input (q, k, v; gate, up) as one product each, for bf16 and Glyd alike, as serving engines run them |
--profile 16 |
GPU time by kernel over this many generated tokens at each --batch size: a step after the prompt, and the prompt’s |
--prefill 64,512,2048 |
prompt lengths to time one forward pass and generate() to its first token at |
--ppl FILE |
perplexity over windows of --ppl-window tokens (default 64) from the file’s 10th MB, and how often the next-token choice is bf16’s |
--mmlu 300 |
MMLU accuracy over this many questions, 0-shot, by the answer letter’s logit |
--smi PREFIX |
once the model is on its GPUs, nvidia-smi’s report into PREFIX-bf16.txt and PREFIX-glyd.txt |
--gpus 2 |
spread the layers over this many GPUs (bf16 by accelerate’s device map, Glyd balanced by packed size) |
--gpu-mem 40 |
GiB of bf16 weights a GPU may hold in the baseline’s device map (default all but 2 GiB) |
--kv 1024,4096 |
prompt lengths (tokens of enwik8, needs --ppl) to generate after with the KV cache compressed, against the plain cache |
--gemm-max 64 |
with fast, steps of up to this many tokens run Glyd’s own products |
Repeating a benchmark
Every number on this site comes from a run of these scripts (in the tags up to v0.26.0), and every run’s log is in the repository: benchmarks/gpu. The Qwen3.8-27B run on an H100 PCIe, for one, holds the machine (machine.txt: the GPU, its memory, the driver), the environment (setup.txt: PyTorch and CUDA), the commands (run.log) and each script’s output. It ran:
python e2e.py models/Qwen3.8-27B --fused --tokens 64 --batch 1,8,32 --prefill 64,128,512,2048 \
--ppl enwik8 --mmlu 300 --merge --format auto --baseline --gpus 1 --profile 16 \
--smi results/smi-Qwen3.8-27B-auto-x1- GPU time a token (benchmarks):
--fused --merge --baseline --batch 1,8,32,64 --profile 16, with--formatfor the layout: a step’s GPU time after the prompt, over 16 steps. - Memory in use (Qwen3-32B on one 48 GB GPU):
--smi PREFIX, with--gpus Nwhere bf16 needs more than one GPU. - Quality (the nineteen models):
--baseline --ppl enwik8 --mmlu 300. - Sizes:
python sizes.py MODEL_DIR ..., every Linear layer’s matrix in both layouts, bit for bit. - The products alone:
python gemm.py MODEL_DIR 1,16,64times every product of one layer against bf16’s;python shapes.py MODEL_DIRGlyd’s product;python check.py model.safetensorspacks, unpacks and compares every tensor;python kv.pychecks the compressed KV cache against the plain one.
The rented-GPU runs use scripts/gpu_lambda.sh: one Lambda Cloud instance a run, terminated at the end, its logs copied into benchmarks/gpu/.