The Glyd codec: levels, modes, the store, bindings

Underneath the GPU work, Glyd is a lossless compression library and command line, written in Rust with a C ABI, for the workloads where storage and read CPU decide the bill: object storage, data lakes, logs and telemetry, backups and versioned exports, RPC payloads, caches. It is a drop-in alternative to LZ4, Snappy and zstd, and does two things they do not: it turns record-shaped data (logs, dumps, CSV, JSON lines) into typed columns before compressing (-r), and it compresses a new version of an object against the old one (--base). Install it with Homebrew, cargo or a wheel (getting started).

Levels

Level	Use it for	How it works
`--turbo` (`-t`)	data read far more often than written: caches, assets, KV-cache paging	v6 format, minimum match 10: fewest tokens, one 32-byte copy a token
default	the LZ4 and Snappy slot, with a better ratio and faster reads	v6 format, an LZAV-class finder, minimum match 7
`--fast` (`-1`)	LZ4-class compression speed	v6 format, an LZ4-class finder, minimum match 5
`--max` (`-9`)	the zstd -3 slot: fewer bytes, 3 to 7 times faster reads on a server	v9 format: 8-way interleaved Huffman literals, tANS sequences with repeat offsets, a double-fast parse over an 8 MB window
`--max --long` (`-L`)	events, logs, anything that repeats itself across a file	the same, with a 128 MB long-distance matcher run once before the parse
`--max --dense` (`-D`)	objects written once and read rarely	`--long` in 128 MB units parsed in stripes on all cores
`--ultra` (`-19`)	write once, read many: datasets, release assets	v9 format on an optimal parse, every position priced in the coder’s own bits
`--cold` (`-C`)	stored for years, read rarely: archives, the last copy	context mixing: every bit predicted from eleven contexts, 1 to 1.3 MB/s a core each way

All levels write one container, and the decoder reads any mix; the level and mode are in the stream.

Modes

Record mode, -r, for logs of any shape, SQL dumps, CSV and JSON lines: it detects the shape, turns each field, key path or template slot into a typed stream (integer, decimal and date-time deltas, dictionaries, text), compresses those with the level and rebuilds the bytes exactly.
Base mode, --base, for versions: nightly dumps, snapshots, images, source trees. Each 32 MB of the new version is parsed with the region of the old one that holds its content as history; decoding needs the same base.

glyd --max  events.json -o events.glyd            # the zstd -3 slot: fewer bytes, 3-7x faster reads
glyd --max -r access.log -o access.glyd           # record mode: logs, dumps, CSV, JSON lines as columns
glyd --ultra -r dump.sql -o dump.glyd             # fewest bytes from a parse; slow to write
glyd --cold -r dump.sql -o dump.glyd              # fewest bytes of all; 1 MB/s per core each way
glyd --base dump-mon.sql dump-tue.sql -o tue.glyd # base mode: Tuesday's dump against Monday's
glyd -d --base dump-mon.sql tue.glyd -o tue.sql   # decoding a base-mode file needs the base
glyd -d events.glyd -o events.json                # the level and mode are in the stream
glyd -b bigfile                                   # benchmark every level on your data

The store

glyd-store keeps objects compressed across a bucket: an object is stored as a delta against the stored object it most resembles, found by fingerprints, when that pays, and alone otherwise; chains are at most four long. The objects’ bytes go to a directory or to S3 (or any S3-compatible service); every object reads back byte-exact.

glyd-store bucket/ --put mon.tar tue.tar wed.tar  # the store finds each object's base itself
glyd-store bucket/ --get 2 -o wed.tar             # also --find NAME, --delete ID, --rebase ID, --compact, --verify, --stats
glyd-store meta/ --s3 s3://bucket/prefix --put wed.tar   # objects in S3
glyd-store --audit s3://bucket/prefix             # what the store would save there, from a sample

The store is under the Business Source License 1.1 (licenses).

From code

Rust:

let mut out = Vec::new();
glyd::compress_parallel_into_max(&input, &mut out);   // or compress_into_max, _ultra, _cold, compress_into (default)
let back = glyd::decompress_parallel(&out)?;          // any level, any block mix
glyd::compress_records_into_max(&log, &mut out);      // record mode (-r); decompress() reads it
glyd::compress_with_base(&old, &new, &mut out, false);// base mode; decompress_with_base(&old, &out)

Python (bindings/python):

import glyd
c = glyd.compress(data, records=True)   # decompress(c); pack(objects); Store("bucket/").put(name, data)

C, C++ and Go (cgo over the same header, bindings/go): link libglyd and include include/glyd.h.

size_t cap = glyd_max_compressed_len(n);
int64_t clen = glyd_compress_max_parallel(src, n, dst, cap);   // or glyd_compress / _parallel
int64_t dlen = glyd_decompress_parallel(dst, clen, out, n);

The formats are specified in docs/spec.md; every release decodes every earlier format. The codec’s benchmarks, against zstd, LZ4, xz and brotli on public data, are in the repository’s README.