Glyd v0.24.0: generate() compiled by default, faster Hopper prompts

Glyd v0.24.0 runs generate() compiled by default, with no code change: on an RTX 4080 SUPER, Qwen3-1.7B generates about twice the tokens a second. It also runs Hopper’s prompts faster, fixes the CUDA 12 library on Hopper, and runs an A10’s long prompts faster. Every weight still decodes to its exact bf16 value.

pip install -U "glyd[gpu]"

generate(), compiled by default

On PyTorch 2.13 or later, generate() on a model from glyd.from_pretrained runs compiled, as generate(..., cache_implementation="static") asks transformers to run it: a static cache and the forward pass in a CUDA graph. Plain generate(), 128 new tokens, on an RTX 4080 SUPER:

v0.23.0’s log · v0.24.0’s log

• The first call compiles. Qwen3-8B’s took 17.5 s with PyTorch’s compile caches empty, 6.7 s in a later process: warm up with one short generate() before serving.
• exact=True is never compiled and stays bit-identical to bf16.
• To run it eager, as before: compile=False in glyd.from_pretrained or glyd.gpu.compress, or GLYD_COMPILE=0; for one call, generate(..., disable_compile=True). Below PyTorch 2.13 it stays eager.
• Greedy tokens compiled can differ from the eager loop’s, as a compiled bf16 model’s can from its eager ones.

What else runs eager, and when: getting started.

Prompts on Hopper

Prompts of 129 to 1,024 tokens on Hopper are faster. On an H100 SXM, Qwen3-8B’s forward pass over 1,024 tokens takes 45.0 ms against bf16’s 36.9, still slower (the known gaps); work on datacenter prompts is under way. The log

The CUDA 12 library, which the wheels load for PyTorch built for CUDA 12, had a slower path on Hopper, in v0.22.0 and v0.23.0 too. It no longer does: its products now take as long as the CUDA 13 library’s, with the same outputs, bit for bit.

Long prompts on an A10

An A10’s long prompts are faster. Qwen3-8B’s forward pass, over bf16’s time:

The log

The C API

The CUDA library’s C API is version 3 (the CUDA library).

Where Glyd is still slower than bf16: the known gaps. Every change, with its numbers: the changelog and the release notes.