The Glyd codec: levels, modes, the store, bindings

Underneath the GPU work, Glyd is a lossless compression library and command line, written in Rust with a C ABI, for the workloads where storage and read CPU decide the bill: object storage, data lakes, logs and telemetry, backups and versioned exports, RPC payloads, caches. It is a drop-in alternative to LZ4, Snappy and zstd, and does two things they do not: it turns record-shaped data (logs, dumps, CSV, JSON lines) into typed columns before compressing (-r), and it compresses a new version of an object against the old one (--base). Install it with Homebrew, cargo or a wheel (getting started).

Levels

Level	Use it for
`--turbo` (`-t`)	data read far more often than written: caches, assets, KV-cache paging
default	the LZ4 and Snappy slot, with a better ratio and faster reads
`--fast` (`-1`)	LZ4-class compression speed
`--max` (`-9`)	the zstd -3 slot: fewer bytes, 3 to 7 times faster reads on a server
`--max --long` (`-L`)	events, logs, anything that repeats itself across a file
`--max --dense` (`-D`)	objects written once and read rarely
`--ultra` (`-19`)	write once, read many: datasets, release assets
`--cold` (`-C`)	stored for years, read rarely: archives, the last copy; 1 to 1.3 MB/s a core each way

All levels write one container, and the decoder reads any mix; the level and mode are in the stream. The default, --fast and --turbo write format v6; --max, --max --long, --max --dense and --ultra write format v9.

Modes

Record mode, -r, for logs of any shape, SQL dumps, CSV and JSON lines: it turns the records into typed columns before compressing and rebuilds the bytes exactly.
Base mode, --base, for versions: nightly dumps, snapshots, images, source trees. A new version is compressed against the old one; decoding needs the same base.

glyd --max  events.json -o events.glyd            # the zstd -3 slot: fewer bytes, 3-7x faster reads
glyd --max -r access.log -o access.glyd           # record mode: logs, dumps, CSV, JSON lines as columns
glyd --ultra -r dump.sql -o dump.glyd             # fewest bytes from a parse; slow to write
glyd --cold -r dump.sql -o dump.glyd              # fewest bytes of all; 1 MB/s per core each way
glyd --base dump-mon.sql dump-tue.sql -o tue.glyd # base mode: Tuesday's dump against Monday's
glyd -d --base dump-mon.sql tue.glyd -o tue.sql   # decoding a base-mode file needs the base
glyd -d events.glyd -o events.json                # the level and mode are in the stream
glyd -b bigfile                                   # benchmark every level on your data

The store

glyd-store keeps objects compressed across a bucket: an object is stored as a delta against the stored object it most resembles when that pays, and alone otherwise. The objects’ bytes go to a directory or to S3 (or any S3-compatible service); every object reads back byte-exact.

glyd-store bucket/ --put mon.tar tue.tar wed.tar  # the store finds each object's base itself
glyd-store bucket/ --get 2 -o wed.tar             # also --find NAME, --delete ID, --rebase ID, --compact, --verify, --stats
glyd-store meta/ --s3 s3://bucket/prefix --put wed.tar   # objects in S3
glyd-store --audit s3://bucket/prefix             # what the store would save there, from a sample

The store is under the Business Source License 1.1, as all of Glyd is from v0.29.0 (license).

Models: glyd pack and glyd verify

The same glyd command packs a model’s weights for the GPU and checks them: from v0.27.0 these are the Python tool’s commands, passed to it as glyd run is, on a machine with an NVIDIA GPU and pip install "glyd[gpu]" (getting started).

glyd pack Qwen/Qwen3.5-9B qwen3.5-9b-glyd            # a bf16 checkpoint, packed and saved (--layout mma12: the 12-bit layout)
glyd verify qwen3.5-9b-glyd                          # every tensor decoded and its sha256 checked (--device cuda:0: on the GPU)

They are part of the glyd-gpu package, under the Business Source License 1.1 (license).

From code

Rust:

let mut out = Vec::new();
glyd::compress_parallel_into_max(&input, &mut out);   // or compress_into_max, _ultra, _cold, compress_into (default)
let back = glyd::decompress_parallel(&out)?;          // any level, any block mix
glyd::compress_records_into_max(&log, &mut out);      // record mode (-r); decompress() reads it
glyd::compress_with_base(&old, &new, &mut out, false);// base mode; decompress_with_base(&old, &out)

Python (bindings/python):

import glyd
c = glyd.compress(data, records=True)   # decompress(c); pack(objects); Store("bucket/").put(name, data)

C, C++ and Go (cgo over the same header, bindings/go): link libglyd and include include/glyd.h.

size_t cap = glyd_max_compressed_len(n);
int64_t clen = glyd_compress_max_parallel(src, n, dst, cap);   // or glyd_compress / _parallel
int64_t dlen = glyd_decompress_parallel(dst, clen, out, n);

The formats are specified in docs/spec.md; every release decodes every earlier format. The codec’s benchmarks, against zstd, LZ4, xz and brotli on public data, are in the repository’s README.