Getting started with Glyd: the GPU harness and the codec

Where things stand

Glyd on the GPU runs from its PyTorch harness, the repository’s gpu/ folder. e2e.py packs a model’s weights, checks that every one decodes to its bf16 value, and runs bf16 and Glyd side by side on your GPU; sizes.py measures every matrix of a model in both layouts. There is no package for it yet: pip install "glyd[gpu]", with glyd.from_pretrained() loading models already packed, is in progress (what’s next). The codec underneath installs today (below).

What you need

An NVIDIA GPU and Linux on x86_64. The runs on this site used an RTX 4080 SUPER, an A10, an A100, H100s and RTX A6000s (the benchmarks).
A model released in bf16, which is most models you can run on one GPU (which fit yours).
Nothing else installed: gpu/setup_env.sh sets up uv, Python 3.12, PyTorch built for CUDA 13 and nvcc pinned to the CUDA PyTorch was built with, without root. Glyd’s CUDA extension builds the first time it is imported.

1. The code and the environment

git clone https://github.com/surya-koritala/Glyd
cd Glyd
bash gpu/setup_env.sh ~/gpuenv   # uv, Python 3.12, PyTorch (CUDA 13), nvcc; no root
source ~/gpuenv/cuda.sh          # the environment's python and nvcc first on PATH

The script ends by printing PyTorch’s version, its CUDA, the number of GPUs and the first one’s name, then nvcc’s build: on the H100 PCIe run, torch 2.14.0+cu130 CUDA 13.0 1 GPUs NVIDIA H100 PCIe. Then build the extension and check the products on your GPU:

cd gpu
python glyd_gpu.py

Each line names a kernel and a test matrix and says whether its products matched a reference within 1e-2 from 1 to 600 tokens and came out the same every run, for example mma_gemm_wg 3072x5120, 238777 exceptions: 1-600 tokens within 1e-2, the same every run.

2. A model

Any model released in bf16, in Hugging Face’s layout. The runs download them with the hf command the environment installs:

hf download Qwen/Qwen3-8B --local-dir ~/models/Qwen3-8B

Gated models (Llama, Gemma) need a Hugging Face token, or the ungated copies under unsloth/ that the runs used: the same weights.

3. Pack it and run it beside bf16

python e2e.py ~/models/Qwen3-8B --format auto --fused --merge --baseline --batch 1,8,32

--format auto takes the layout for your GPU, --fused multiplies straight from the packed weights, --merge runs q, k, v and gate, up as one product each as serving engines do, --baseline runs bf16 first, and --batch the sequences at once to generate for. Add --profile 16 for GPU time a token, --ppl enwik8 for perplexity and --mmlu 300 for MMLU; every option is on the GPU page. The text the quality checks use:

curl -sL http://mattmahoney.net/dc/enwik8.zip -o enwik8.zip
python -c "import zipfile; zipfile.ZipFile('enwik8.zip').extractall('.')"

What it prints, from the Qwen3.8 27B run on an H100 PCIe with all of those, a line for each size and measure (trimmed to one of each):

bf16 (weights 53.79 GB): batch 1: 9.7 tokens/s (9.7 a sequence), peak VRAM 54.01 GB
bf16 perplexity: 15.1946 (12600 tokens)
auto: mma12, the 12-bit decode keeps up with this GPU's memory
mma12 fused: packed in 12 s; weights 40.35 GB against 53.79 GB bf16 (75.0%), scratch 0.27 GB, VRAM in use 40.65 GB on 1 GPU
glyd mma12 fused: batch 1: 12.3 tokens/s (12.3 a sequence), peak VRAM 40.84 GB
glyd mma12 profile, batch 1: 92.53 ms a step, GPU busy 34.73 ms a step after the prompt (29 tokens/s of GPU time), 50.26 ms for the prompt and a token
glyd mma12 perplexity: 15.1941 (12600 tokens)
glyd mma12 MMLU: 80.00% of 300 questions (0-shot)
next-token choice as bf16's: 98.82%
MMLU answer as bf16's: 99.67%
logits bit-identical: False
generated tokens identical to bf16: 13 of 64

bf16 (weights …): the model as released, generating for each --batch size: tokens a second and its peak GPU memory.
auto: …: the layout --format auto took for this GPU, and why.
packed in …: Glyd’s weights against bf16’s (here 75.0%, the 12-bit layout), and the GPU memory in use.
glyd … batch 1: the same generation from the packed weights.
profile: with --profile N, the wall time of a step, the GPU’s busy time a step after the prompt (the GPU time a token this site shows) and the prompt’s; then the kernels that took it.
next-token choice as bf16's, MMLU answer as bf16's: how often Glyd’s model picks bf16’s token or answer.
logits bit-identical: False and generated tokens identical … 13 of 64: the weights are the same to the bit, but the products add their terms in another order than cuBLAS’s, so the logits differ by a rounding and a greedy generation parts from bf16’s at the first close call. That is also why the quality lines move a hair in either direction.

4. Every matrix, both layouts

python sizes.py ~/models/Qwen3-8B

It packs every Linear layer’s matrix in the tiered layout and in the 12-bit layout, unpacks it and compares it with the original bit for bit, and reports the matrices’ size in bf16 and in each layout, in bits a weight and GB: the tables of the nineteen-model report. Several model directories can follow one another.

The codec

The lossless codec under the GPU work installs as a package:

pip install glyd                             # Python: Linux x86_64 / aarch64 (glibc 2.35+), macOS arm64
brew install surya-koritala/glyd/glyd        # macOS / Linux: the glyd and glyd-store CLIs, glyd.h
cargo install --git https://github.com/surya-koritala/Glyd glyd glyd-store   # from source

The Python package is on PyPI. Every release also carries the command-line tools, the shared and static libraries and glyd.h for the same platforms. In Python:

import glyd
packed = glyd.compress(data)            # the --max level
assert glyd.decompress(packed) == data

From the command line, a first file:

glyd --max events.json -o events.glyd   # the zstd -3 slot: fewer bytes, faster reads
glyd -d events.glyd -o events.json      # the level and mode are in the stream

Its levels, modes, store and bindings: the codec.