Unsloth Dynamic (UD) GGUFs

Unsloth Dynamic (UD) is Unsloth's selective-layer quantization format for GGUF models. Important layers are upcast to 8/16-bit while less critical layers are aggressively quantized, enabling massive models (671B+) to run on consumer hardware.

Variants

Variant	Bit-width	Description	Use Case
`UD-TQ1_0`	~1-bit	Dynamic 1-bit, extreme compression	Minimal RAM (e.g., DeepSeek-V3.1 671B → ~192 GB)
`UD-IQ2_M`	~2-bit	Importance-weighted 2-bit; slower conversion, may have better accuracy	Accuracy-maximizing 2-bit
`UD-Q2_K_XL`	~2.7-bit	Recommended balance of size vs. accuracy; faster conversion than IQ2_M	General purpose (e.g., GLM-5.1 → ~220 GB)
`UD-Q3_K_XL`	~3-bit	Dynamic 3-bit	Middle ground
`UD-Q4_K_XL`	~4-bit	Dynamic 4-bit, highest quality	When RAM is plentiful

Download

# Dynamic 2-bit (recommended)
hf download unsloth/GLM-5.1-GGUF \
    --local-dir unsloth/GLM-5.1-GGUF \
    --include "*UD-Q2_K_XL*"

# Dynamic 1-bit
hf download unsloth/GLM-5.1-GGUF \
    --local-dir unsloth/GLM-5.1-GGUF \
    --include "*UD-TQ1_0*"

Python:

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="unsloth/GLM-4.7-GGUF",
    local_dir="unsloth/GLM-4.7-GGUF",
    allow_patterns=["*UD-Q2_K_XL*"],
)

Compatibility with 1bit.MONSTER

A UD file is an ordinary GGUF. "UD" names Unsloth's per-tensor recipe, not a new quant type: every tensor carries a standard ggml type, and the recipe picks a different one per tensor. For example, the tensor types in unsloth/Qwen3-0.6B-GGUF, read with gguf-py on 2026-09-23:

File	Tensor types inside
`UD-Q4_K_XL`	Q6_K (attn_v, ffn_down, token_embd), Q5_K, Q4_K, and IQ4_XS on about 5 layers
`UD-Q2_K_XL`	Q2_K, Q3_K, IQ2_S/IQ2_XS/IQ3_S/IQ3_XXS/IQ4_XS on about 5 layers; token_embd kept at Q6_K
`UD-IQ2_M`	IQ2_S, IQ3_S, IQ3_XXS, IQ2_XS, IQ4_XS; token_embd Q5_K
`UD-IQ1_S`	IQ1_S, IQ1_M, IQ2_XXS, IQ2_S, IQ3_S, IQ3_XXS, Q2_K; token_embd Q5_K

So whether a UD file loads depends only on the ggml types in it, and every UD file uses some IQ types.

Reader	UD files
llama.cpp: the HRX + Vulkan build in 1bit-MONSTER/engine (`ggml_vulkan`, HRX)	Load and run. All of the types above are decoded
The project reader (`src/gguf_reader.cpp`: `cpu_generic`, `gguf_to_onebp`)	Refused with a clear error. It decodes the block quants and ternary `TQ1_0`/`TQ2_0` (#2057), but no IQ type

Measured on Strix Halo on 2026-09-23. The model is Qwen3-0.6B, run with the engine's llama.cpp fork (engine main b11258e). The metrics are llama-perplexity --kl-divergence over 20 x 512 tokens against BF16 (PPL 10.47), and llama-bench tg128:

File	Size	PPL	Mean KLD	Same top token	Vulkan0 tok/s	HRX20 tok/s
BF16	1198 MB	10.47	-	-	115	40
Q4_K_M	397 MB	10.82	0.063	89.1%	306	68
UD-Q4_K_XL	405 MB	10.68	0.052	89.8%	~300	83
UD-Q2_K_XL	302 MB	16.23	0.662	66.3%	337	92
UD-IQ2_M	269 MB	20.44	0.857	64.0%	357	62
UD-IQ1_S	215 MB	10633	7.41	6.0%	374	89

At 4 bits, UD-Q4_K_XL beats Q4_K_M. Its KL is 17% lower for 2% more bytes, at the same Vulkan speed. A first run of UD-Q4_K_XL showed 202 tok/s; 5 repetitions per round measured 289-302 against Q4_K_M's 306-307.
Below 4 bits, a model this small loses a lot. The 1-bit and 2-bit UD recipes are aimed at 100B+ models, where the size saving matters and the quality loss is smaller.
Routing. Send UD files to the llama.cpp backends. On this box Vulkan0 is 3-5x faster than HRX20 for every file above.

Where to Find Them

All UD quantized models live on Hugging Face under the unsloth/ org, in repos named *GGUF. Filename pattern: *UD-<variant>*:

unsloth/GLM-5.1-GGUF — *UD-Q2_K_XL*, *UD-TQ1_0*, *UD-IQ2_M*
unsloth/GLM-4.7-GGUF — *UD-Q2_K_XL*, *UD-TQ1_0*
unsloth/gemma-4-26B-A4B-it-GGUF — *UD-Q4_K_XL*
unsloth/DeepSeek-V3.1-GGUF — *UD-Q2_K_XL*, *UD-TQ1_0*

Running with llama.cpp

./llama-cli -m unsloth/.../UD-Q2_K_XL.gguf ...

Reference

Unsloth Dynamic 2.0 GGUFs