Questions and answers
Is this quantization?
No. Quantization rounds weights to fewer levels, which changes the model. Glyd stores the same values in fewer bits and gives every one of them back. It is closer to a zip file than to 4-bit. How it works
Does it change the model's answers?
The weights are the model's to the bit. The matrix products add their terms in another order than cuBLAS does, as any two GPU kernels do, so a close call can go either way: over the models measured, Glyd's MMLU answers are bf16's on 98.0 to 100% of the questions and perplexity stays within 0.06% of bf16's. The quality tables
Is it smaller than 4-bit quantization?
No. 4-bit formats take about 4.5 bits a weight; Glyd takes about 11. Glyd is for when you need the model exactly as it was trained, on less hardware. How it compares
Which GPUs?
Measured on an RTX 4080 SUPER, RTX A6000s, an A10, an A100 and H100s. Glyd picks the layout for the GPU: the tiered one on RTX 40-series (Ada) and wherever only it fits, the 12-bit one on the others. Blackwell GPUs are not measured yet. Which layout on which GPU
Which models?
Any model released in bf16: nineteen popular open models are measured, every matrix 32 to 33% smaller. FP8 models can shrink 16 to 18% (measured on three), but the GPU kernels for them are not built yet; 4-bit weights have little left to take. Every model
How do I install it?
Today Glyd on the GPU runs from its PyTorch harness: clone the repository, run gpu/setup_env.sh, then e2e.py on a model. A Python package, pip install "glyd[gpu]" with glyd.from_pretrained(), is in progress. The codec underneath installs with Homebrew, cargo or a wheel from the releases. Getting started
Is it faster than bf16?
Usually, at a few sequences a step: a generated token reads every weight once, and Glyd reads a third fewer bytes. On an RTX 4080 SUPER it takes 9 to 28% less GPU time a token for Qwen2.5-7B from 1 to 64 sequences a step. At 32 to 64 sequences on an A100 or H100 the decode work shows: 6 to 17% more time a token today. The benchmarks
Does it work with vLLM or SGLang?
Not yet. Glyd runs from its own PyTorch harness today; serving through vLLM, as vllm serve MODEL --quantization glyd, is planned. What's next
Does it compress the KV cache too?
Yes, in the harness. The keys and values are bf16 numbers like the weights and as compressible: Qwen2.5-7B's cache for 16,384 tokens takes 651 MB instead of 947, and decodes back bit for bit. On the GPU
Is it open source?
The codec is open source: BSD-3-Clause OR GPL-2.0, as zstd is. The GPU kernels and the store are source-available under the Business Source License 1.1: free to read, modify and use for personal, educational, research and other non-commercial work, commercial production use needs a license, and each version becomes Apache-2.0 four years after its release. The licenses