Getting started

Two ways in. One command installs Glyd with vLLM and PyTorch, and glyd run downloads a model, packs it as it loads and opens a chat. In Python, pip install "glyd[gpu]" installs Glyd for the GPU, and glyd.from_pretrained() loads a model released in bf16 with its weights packed on the GPU as they arrive, every one decoding to its exact bf16 value. The current release is v0.32.1, of Oct 17, 2026 (the changelog). The harness this site’s benchmarks were measured with is further down, and the codec underneath at the end.

One command

curl -LsSf https://getglyd.com/install.sh | sh
glyd run Qwen/Qwen3.5-9B

The script installs uv where there is none, then Glyd with vLLM 0.30 and PyTorch as one isolated tool, on a Python 3.12 that uv fetches: about 8 GB, all under your home directory, with no sudo, no virtual environment and no pip. It ends with glyd doctor, which says what the machine can run, and the script is one file you can read first. glyd run downloads the model (Qwen3.5-9B is 19.3 GB), checks that this machine can run it, starts vLLM with settings it works out from the GPU, and opens a chat in the terminal and at http://localhost:8000. glyd serve MODEL leaves the server up as an OpenAI API for other programs.

• It needs Linux on x86_64, an NVIDIA GPU of the Ampere generation or newer (RTX 30 and 40 series, A10, A100, L4, H100 and later) and its driver, 580 or newer, curl, and 10 GB of free disk in your home directory besides the model’s. No CUDA toolkit and no sudo.
• On a Mac, or a Linux machine with no NVIDIA GPU, glyd run has nothing to run on. The script says so, and installs what works there: the compression program (glyd FILE -o OUT), the release’s download checked against its sha256 (the codec).
• On a 24 GB card, measured on an RTX 4090 with vLLM 0.30.0 (v0.29.0): Qwen3.5-9B with Glyd took 12.68 GiB of weights and glyd run runs it with a 189,440-token context; bf16’s weights, 17.66 GiB, did not start with vLLM’s default settings (out of memory at a 4,096-token context). On an RTX 5090 Glyd’s weights took 13.89 GiB (the run’s record).
• On a 16 GB card (v0.26.0), measured on an L4 held to what an RTX 4080 SUPER with a desktop leaves free (it was not run on that card): Qwen3-8B with Glyd took 11.39 GiB of weights and glyd run chose an 11,264-token context; bf16’s weights, 15.26 GiB, left no room, and its server did not start with the same flags. The settings, what the script does to a machine, Open WebUI and the numbers: local chat, like Ollama.
• The KV cache: glyd run keeps vLLM’s own. GLYD_KV=auto glyd run MODEL holds it in fewer bits, every value read back bit for bit, on an A100, an L4, an H100 and a GH200 (any other GPU keeps vLLM’s own, with a line that says so); the first start with it takes minutes (on an L4, 130 s for an 8,192-token window and 217 s for Qwen3-8B’s whole 40,960-token window), once for the model. Qwen3.5 and Qwen3.8 keep vLLM’s own cache: the lossless one does not hold linear attention or a head size of 256 yet. glyd serve and vllm serve MODEL --quantization glyd use auto already; GLYD_KV=off keeps vLLM’s own there. The options and the numbers: the lossless KV cache.
• Fewer bits, levels (v0.29.4): glyd run MODEL:LEVEL sets how many bits the weights keep on the GPU. :penguin is lossless, the default; :kestrel is about 6.5 bits per weight (--lossy, from v0.29.3, still works); :swift is about 5.5. Kestrel and swift are not lossless and run on Glyd’s own engine only (Qwen3.5, Qwen3.6 and Qwen3.8 dense models), one user at a time, in the terminal; a level that is not built yet is refused with the list of those that are. Measured with Qwen3.8-27B on one A100 80GB, a 1,024-token prompt and 256 more: :swift took 17.4 GiB of weights and ran at 47.5 tokens a second, :kestrel 20.5 GiB and 46.3. Over 253,952 positions of WikiText-2 the next-token distribution differs from bf16’s by a KL divergence of 0.00477 for :kestrel (0.00465 for the FP8 checkpoint) and 0.01393 for :swift. Starting takes longer, as the weights are fitted more closely while they load: 127 s for kestrel and 157 s for swift on the A100. glyd --help shows the levels.

What you need

For glyd run, as above. For the Python package:

• Linux on x86_64 or aarch64, glibc 2.28 or later (the wheels are manylinux_2_28). Windows and macOS have no GPU package; the codec runs there (below).
• An NVIDIA GPU, Ampere or later: RTX 30 and 40 series, A10, A100, L40S, H100 and the like. Of the Blackwell GPUs, the RTX PRO 6000 and the RTX 5090 are measured; the B200 is not yet. Every GPU a release ran on: tested GPUs.
• PyTorch 2.5 or later built for CUDA 12 or 13; pip installs one if there is none. generate() runs compiled by default from PyTorch 2.13 on, eager below it.
• A model released in bf16, which is most models you can run on one GPU (which fit yours). FP8 and 4-bit checkpoints load as they are, with little to gain (why).

Install

pip install "glyd[gpu]"

The gpu extra installs torch>=2.5, transformers>=5.17,<6, accelerate>=1.1, safetensors>=0.8 and huggingface_hub>=1.5,<2. From v0.27.0 the GPU half is the glyd-gpu package, which the extra brings (the same version, compiled wheels): it carries the GPU kernels prebuilt for CUDA 12.8 and 13.0, libglyd_gpu_cuda12.so and libglyd_gpu_cuda13.so, and loads the one for PyTorch’s CUDA: no compiler, no clone. GLYD_GPU_LIB names another build. The same kernels come on their own for engines in C, C++ or Rust: the CUDA library.

To serve with vLLM in a setup of your own, pip install "glyd[vllm]", then vllm serve MODEL --quantization glyd (serving with vLLM).

Load a model and generate

import glyd
from transformers import AutoTokenizer

model = glyd.from_pretrained("Qwen/Qwen3.5-9B")          # any bf16 checkpoint, packed on the GPU as it loads
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B")
out = model.generate(**tok("The history of data compression", return_tensors="pt").to(model.device), max_new_tokens=64)
print(tok.decode(out[0]))

glyd.from_pretrained(name_or_path, *, device="cuda:0", layout="auto", exact=False, merge=True, verify=False, compile=True, **hf_kwargs) returns the transformers model, ready for generate(): a causal LM, or an image-text-to-text model that loads only with its vision tower. transformers loads the checkpoint, and every Linear’s weight is packed on the GPU as it arrives.

• layout: "auto" takes the faster layout for the GPU: "mma", the smallest (10.8 bits a weight), on RTX 40-series (Ada), and for a mixture of experts on an A10 too, and "mma12", the 12-bit layout, on A10, A100, H100 and the RTX 50 series (which layout on which GPU). On an L4 or L40S, "mma12" gives the fastest short prompts, at 25% less memory against "mma"’s 33%.
• merge: q, k, v and gate, up as one product each, as serving engines run them.
• exact: the logits bit for bit bf16’s (below).
• verify: every tensor of a saved model decoded and its sha256 checked (below).
• compile: generate() compiled, the default on PyTorch 2.13 or later; False runs it eager (below).
• hf_kwargs: passed to transformers: revision, token (gated models such as Llama and Gemma need one), device_map for several GPUs, attn_implementation and the rest.

glyd.gpu.compress(model) packs a bf16 model already loaded, in place.

Compiled generation

On PyTorch 2.13 or later, generate() runs compiled by default, with no code change, as generate(..., cache_implementation="static") asks transformers to run it: a static cache, and the forward pass under torch.compile with CUDA graphs. Plain generate(), 128 new tokens, on an RTX 4080 SUPER (v0.24.0, Sep 2026):

v0.23.0’s log · v0.24.0’s log

• The first call compiles. Qwen3-8B’s took 17.5 s with PyTorch’s compile caches empty, 6.7 s in a later process: warm up with one short generate() before serving (a streamer’s consumer waits through it).
• Eager, as before: compile=False in glyd.from_pretrained or glyd.gpu.compress, or GLYD_COMPILE=0; for one call, generate(..., disable_compile=True). exact=True is never compiled.
• Eager anyway: a family transformers does not compile whole, a model over several GPUs, and Llama 4; a call with several beams, an assistant or another assisted mode, its own cache or a cache_implementation, use_cache=False, return_dict_in_generate, attentions or hidden states, or custom_generate; and a call whose static cache would hold more than 1,280 positions in all on a GeForce card, 2,048 on another (GLYD_COMPILE_MAX sets it), where the eager loop is as fast.
• Greedy tokens compiled can differ from the eager loop’s, as a compiled bf16 model’s can from its eager ones.

Exact mode and what the default does

model = glyd.from_pretrained("Qwen/Qwen3.5-9B", exact=True) # logits bit-identical to bf16's

The weights are bf16’s to the bit either way; what differs is the arithmetic. By default the products round a little differently from bf16’s, so the logits differ from bf16’s by a rounding. With exact=True the products are bf16’s own, so the logits are bf16’s bit for bit. It is slower than the default, and never compiled: its tokens are bf16’s eager ones.

Neither is exactly the fp32 result. Against the same model run in fp32, at the prompt’s last position, on an RTX 4080 SUPER:

gpu/check_models.py (in v0.26.0’s tag) makes the same checks against bf16 and the fp32 model, with exact mode and save and verify; nine dense models from Qwen3-0.6B to Llama-3.1-8B-Instruct pass them (how it was measured).

Save it packed and verify

glyd.save_pretrained(model, "qwen3-8b-glyd")               # the packed format, loads without packing again
model = glyd.from_pretrained("qwen3-8b-glyd", verify=True) # decodes every tensor, checks its sha256

glyd.save_pretrained(model, path) writes the packs as safetensors and glyd.json: the format, the source, the layout and every tensor’s shape and the sha256 of its original bytes, the tensors saved as they are (norms, biases, embeddings) included. The config and tokenizer files are copied beside them. With verify=True every tensor is decoded and checked against its sha256.

On other GPUs a save is packed again in their layout as it loads (which layout on which GPU). A 12-bit save loaded as saved skips the packing; on an RTX 4080 SUPER with layout="mma12", three processes each:

The log. The 12-bit layout’s bytes changed in v0.25.0: pack a 12-bit model from v0.24.0 or before again; mma saves are unchanged.

The same from Python’s command line:

python -m glyd_gpu pack Qwen/Qwen3-8B qwen3-8b-glyd   # packed, checked and saved (--layout mma12: glyd-v3)
python -m glyd_gpu verify qwen3-8b-glyd               # every tensor decoded, its sha256 checked

Pack and verify from the command line

glyd pack and glyd verify are the commands of the Python tool, as glyd run is (the glyd program passes them to it): the same files as python -m glyd_gpu pack writes, in either layout, on a machine with an NVIDIA GPU and pip install "glyd[gpu]".

glyd pack Qwen/Qwen3-8B qwen3-8b-glyd                # a directory, or a model in the Hugging Face cache
glyd pack Qwen/Qwen3-8B qwen3-8b-12 --layout mma12   # the 12-bit layout (glyd-v3)
glyd verify qwen3-8b-glyd                            # every tensor decoded, its sha256 checked; --device cuda:0 on the GPU

• Before v0.27.0 the Rust program packed on the CPU, with no Python, PyTorch or GPU, in the families Qwen3, Qwen2, Llama, Mistral, Granite and GraniteMoe: Qwen3-8B’s 16.4 GB in 18.5–23.0 s in mma and 22.8–24.6 s in mma12, at the pace of the disk (the log). The glyd-gpu program and crate are gone from v0.27.0, and the release’s files and Homebrew carry glyd and glyd-store.
• Where: the commands are part of the glyd-gpu package, under the Business Source License 1.1, as all of Glyd is from v0.29.0 (license).

Will it fit

print(glyd.fit("Qwen/Qwen3.5-27B", gpu="48GB"))            # will it fit, bf16 against Glyd

glyd.fit(name_or_path, gpu="48GB", context=8192) answers from config.json and the checkpoint’s metadata, without downloading the weights, with Glyd’s measured sizes where it has them. The rule is the one the GPU pages use: the weights, an 8K-token KV cache and 1.5 GiB for the runtime, against the memory nvidia-smi reports. From the command line:

python -m glyd_gpu fit Qwen/Qwen3-1.7B --gpu 16GB

Qwen3-1.7B on a 16 GB GPU: bf16 needs 6.6 GB, fits; Glyd 5.3 GB, fits

Several GPUs

device_map="auto" goes through to transformers and spreads the layers over the GPUs it sees: glyd.from_pretrained("Qwen/Qwen3-32B", device_map="auto").

Mixtures of experts

The same glyd.from_pretrained() packs a mixture of experts, every family of transformers 5.17 (OLMoE, granite MoE, Qwen3-MoE, Qwen3-Next, Gemma 4, GLM-4.5, Mixtral, gpt-oss and DeepSeek V3 among them). With exact=True their logits are bf16’s bit for bit. On an RTX 4080 SUPER, one sequence:

Llama 4, DBRX, Aria, JetMoE, Step 3.7 and LongCat-Flash are packed too; Llama 4’s experts stayed bf16 before v0.22.0. A Llama-4-Scout-17B-16E-Instruct MoE block at its real sizes (hidden 5120, 16 experts of 8192) on an RTX 4080 SUPER, bit for bit with exact=True (v0.22.0):

save_pretrained saves a mixture of experts with its packs; such a checkpoint’s format is glyd-v4 from v0.28.0 (glyd-v2 before, which glyd 0.21 refuses; a dense model’s was glyd-v1). from_pretrained(path) loads it, verify=True checks every tensor, and python -m glyd_gpu pack and verify take one. granite-3.1-3b-a800m on an RTX 4080 SUPER, its logits bit for bit those of the model packed as it loads (v0.22.0):

A model with packed experts can’t be copied or pickled (copy.deepcopy, torch.save): load it again.

Not in this release

• Glyd’s own engine on 2026 models: it ships with v0.29.0, off unless asked for, and 2026 models run on vLLM (serving with vLLM).
• The KV cache held in fewer bits for generate(): it is in vLLM only (the lossless KV cache).
• A GPU package for Windows or macOS; the codec runs there (below).
• 33% on every model and GPU, and speed at every load. The v0.29.0 runs below measured weights 21.3 to 28.2% smaller than bf16’s; v0.29.1 compresses additional loaded weights. One-third savings across every model and GPU remain a target. The v0.28.0 H100 run measured each token at full load taking 1.43 times bf16’s time. Earlier measurements remain listed with their release. Each, with its logs: the known gaps.

Reproduce the benchmarks

The benchmarks on this site were measured with Glyd’s PyTorch harness, the repository’s gpu/ folder, which is in the tags up to v0.26.0; from v0.27.0 the GPU code is not in the public repository, and these steps run v0.26.0’s, the last release that carries it. e2e.py packs a model’s weights, checks that every one decodes to its bf16 value, and runs bf16 and Glyd side by side on your GPU; sizes.py measures every matrix of a model in both layouts. Unlike the package, it runs from a clone and builds its own CUDA extension. What it needs:

• An NVIDIA GPU and Linux on x86_64 (tested GPUs).
• A model released in bf16 (which fit yours).
• Nothing else installed: gpu/setup_env.sh sets up uv, Python 3.12, PyTorch built for CUDA 13 and nvcc pinned to the CUDA PyTorch was built with, without root. Glyd’s CUDA extension builds the first time it is imported.

1. The code and the environment

git clone --branch v0.26.0 https://github.com/surya-koritala/Glyd
cd Glyd
bash gpu/setup_env.sh ~/gpuenv   # uv, Python 3.12, PyTorch (CUDA 13), nvcc; no root
source ~/gpuenv/cuda.sh          # the environment's python and nvcc first on PATH

The script ends by printing PyTorch’s version, its CUDA, the number of GPUs and the first one’s name, then nvcc’s build: on the H100 PCIe run, torch 2.14.0+cu130 CUDA 13.0 1 GPUs NVIDIA H100 PCIe. Then build the extension and check the products on your GPU:

cd gpu
python glyd_gpu.py

It prints a line for each product it checks, saying whether the results matched a reference within 1e-2 from 1 to 600 tokens and came out the same every run.

2. A model

Any model released in bf16, in Hugging Face’s layout. The runs download them with the hf command the environment installs:

hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B

Gated models (Llama, Gemma) need a Hugging Face token, or the ungated copies under unsloth/ that the runs used: the same weights.

3. Pack it and run it beside bf16

python e2e.py ~/models/Qwen3-8B --format auto --fused --merge --baseline --batch 1,8,32

--format auto takes the layout for your GPU, --fused runs Glyd’s own products, --merge runs q, k, v and gate, up as one product each as serving engines do, --baseline runs bf16 first, and --batch the sequences at once to generate for. Add --profile 16 for GPU time a token, --ppl enwik8 for perplexity and --mmlu 300 for MMLU; every option is on the GPU page. The text the quality checks use:

curl -sL http://mattmahoney.net/dc/enwik8.zip -o enwik8.zip
python -c "import zipfile; zipfile.ZipFile('enwik8.zip').extractall('.')"

What it prints, from the Qwen3.8-27B run on an H100 PCIe with all of those, a line for each size and measure (trimmed to one of each):

bf16 (weights 53.79 GB): batch 1: 9.7 tokens/s (9.7 a sequence), peak VRAM 54.01 GB
bf16 perplexity: 15.1946 (12600 tokens)
mma12 fused: packed in 12 s; weights 40.35 GB against 53.79 GB bf16 (75.0%), scratch 0.27 GB, VRAM in use 40.65 GB on 1 GPU
glyd mma12 fused: batch 1: 12.3 tokens/s (12.3 a sequence), peak VRAM 40.84 GB
glyd mma12 profile, batch 1: 92.53 ms a step, GPU busy 34.73 ms a step after the prompt (29 tokens/s of GPU time), 50.26 ms for the prompt and a token
glyd mma12 perplexity: 15.1941 (12600 tokens)
glyd mma12 MMLU: 80.00% of 300 questions (0-shot)
next-token choice as bf16's: 98.82%
MMLU answer as bf16's: 99.67%
logits bit-identical: False
generated tokens identical to bf16: 13 of 64

• bf16 (weights …): the model as released, generating for each --batch size: tokens a second and its peak GPU memory.
• packed in …: Glyd’s weights against bf16’s (here 75.0%, the 12-bit layout), and the GPU memory in use.
• glyd … batch 1: the same generation from the packed weights.
• profile: with --profile N, the wall time of a step, the GPU’s busy time a step after the prompt (the GPU time a token this site shows) and the prompt’s; then the kernels that took it.
• next-token choice as bf16's, MMLU answer as bf16's: how often Glyd’s model picks bf16’s token or answer.
• logits bit-identical: False and generated tokens identical … 13 of 64: the weights are the same to the bit, but by default the products round a little differently from bf16’s, so the logits differ by a rounding and a greedy generation parts from bf16’s at the first close call. That is also why the quality lines move a hair in either direction.

4. Every matrix, both layouts

python sizes.py ~/models/Qwen3-8B

It packs every Linear layer’s matrix in mma and in mma12, unpacks it and compares it with the original bit for bit, and reports the matrices’ size in bf16 and in each layout, in bits a weight and GB: the tables of the nineteen-model report. Several model directories can follow one another.

The codec

The lossless codec under the GPU work installs as a package:

pip install glyd                             # Python: Linux x86_64 / aarch64 (glibc 2.28+), macOS arm64
brew install surya-koritala/glyd/glyd        # macOS / Linux: the glyd and glyd-store programs, glyd.h
cargo install --git https://github.com/surya-koritala/Glyd glyd glyd-store   # from source

The Python package is on PyPI. Every release also carries the command-line tools, the shared and static libraries and glyd.h for the same platforms. In Python:

import glyd
packed = glyd.compress(data)            # the --max level
assert glyd.decompress(packed) == data

From the command line, a first file:

glyd --max events.json -o events.glyd   # the zstd -3 slot: fewer bytes, faster reads
glyd -d events.glyd -o events.json      # the level and mode are in the stream

Its levels, modes, store and bindings: the codec.