Glyd v0.32.0: Gemma 4 12B and 31B, lossless

Glyd v0.32.0 is out.

Gemma 4 12B and 31B, lossless. glyd run glyd/gemma-4-12B-penguin downloads glyd/gemma-4-12B-penguin (16.1 GB, plus Google’s 0.88 GB draft model) and starts it. The 31B is glyd/gemma-4-31B-penguin (41.3 GB + 0.97 GB). Every weight is Google’s bf16 weight bit for bit, at about 10.8 bits per weight: the 12B takes 15.0 GiB on the GPU in place of 22.2 GiB and runs on a 24 GB card; the 31B takes 38.4 GiB in place of 57.2 GiB and runs on a 48 GB card. glyd run google/gemma-4-12B-it with no level does the same.

curl -LsSf https://getglyd.com/install.sh | sh
glyd run glyd/gemma-4-12B-penguin
glyd verify glyd/gemma-4-12B-penguin

One user, greedy, with Google’s draft model (without it in brackets):

GPU, model	Chat	Long prompt read	GSM8K	MMLU-Pro
RTX 4090, 12B	89.8 tokens a second (50.8)	28K tokens in 7.6 s	72/75	67/75
RTX A6000 48 GB, 31B	34.3 (13.3)	24K tokens in 35.5 s
H100 PCIe, 31B	61.8 (30.0)	24K tokens in 23.0 s	72/75	67/75

The weights are the same as Google’s, so the only difference in the output is the order of the arithmetic. Against transformers’ float32 run (wikitext-2, 32 x 512) the 12B on Glyd’s engine has a mean KLD of 0.170, where transformers’ own bf16 runs are at 0.132 and 0.143.

Known limits: Gemma 4 runs on Linux with an NVIDIA RTX 30 series card or newer, text only, one request at a time. On H100 the engine isn’t tuned yet: prompt reading runs well below the card’s peak, and the 31B without the draft model writes at about 62% of what the card’s memory bandwidth allows.

Full details are in the release notes and the changelog; what is not done yet is on the known gaps.