Quantizing Qwen3.8-Flash-Next on one unified-memory box

Field notes from fitting Qwen/Qwen3.8-Flash-Next (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB unified memory, aarch64, CUDA 13, sm_121. Measured with llm-compressor 0.13.0, compressed-tensors 0.18.0, transformers 5.16.1, torch 2.11.0+cu130.

Revision note. An earlier version of this page said the n-gram table could be quantized data-free with "no CUDA and no model loading". The first half is wrong and is corrected below: the data-free pipeline still onloads the module to the GPU and dies exactly like calibration does. The file-level route described at the end is the one that works, and its numbers are now measured rather than projected.

The one thing that matters: layer 1 is one 102 GB module

The per-layer n-gram embedding (PLE) lives entirely on decoder layer 1. From the checkpoint index:

• layer 1 holds 161 tensors; every other layer holds 24
• 130 of those are …layers.1.ple.ple_embedding.ngram_embedding.shard_N.weight
• the table is 51.2 B params, ~102 GB in bf16

The critical detail, and the one that took five failed runs to find: those 130 shards are a file-layout artifact. At load time they assemble into a single nn.Embedding. A diagnostic over named_modules() on the loaded model:

ngram Embedding modules: 1  e.g. [...layers.1.ple.ple_embedding.ngram_embedding]
other Embedding modules: 2  e.g. [model.visual.pos_embed, model.language_model.embed_tokens]

So every module-level path has to materialise one 102 GB parameter. On a discrete GPU that is awkward; on unified memory, where host RAM and GPU memory are one pool, it has to fit twice over in 121.7 GB — resident, then onloaded — and it does not.

This is the practical form of the vendor guidance that TP2 is the validated minimum for this model.

Data-free does not mean no GPU

Worth stating plainly because it is the trap that cost the most time. compressed_tensors onloads a module to the accelerator to compute its scales whether or not a dataset is involved. Running oneshot(..., pipeline="datafree") against the PLE table fails with the same driver-level cudaErrorMemoryAllocation as full calibration, after Applying quantization config: 1/1.

"Weight-only and calibration-free" describes the math, not the memory path.

(Also: the registered pipeline name is datafree. The module directory is data_free, and passing that raises KeyError: Unable to find data-free registered under CalibrationPipeline. Registered values: basic, datafree, independent, sequential.)

Parameter census

Measured by instantiating the real config on the meta device and bucketing named_parameters():

Only 12 of 48 layers are full attention (full_attention_interval: 4); the other 36 are linear attention with constant-size state, so KV cache is unusually cheap for the parameter count.

NVFP4 costs 0.5625 bytes/param, not 0.5

Four bits plus one fp8 scale per group of 16 = 0.5 + 1/16. Plans built on 0.5 are ~11% optimistic.

Validated twice. Against a published conversion: 120.796e9 × 0.5625 = 67.95 GB vs its 68.0 GB experts file. And directly, packing one real n-gram shard: 0.800 GB bf16 → 0.225 GB (3.56×), which is 0.5625 bytes/param plus the fp8 scale array.

Published quantizations are all too big, for one reason

The 135.3 GB outlier is the only one that touches the PLE table at all (fp8, 102.4 → 51.2 GB); everything else leaves it bf16, which is the ~180 GB floor. Even the best exceeds 121.7 GB before any KV cache.

llm-compressor on one unified-memory GPU: six traps

Each of these cost a multi-hour run. All are specific to device_map="auto_offload" on a single GPU.

1. load_context() is the loader, not load_quantizable_moe(). auto_offload is not a transformers device map; it is legal only because compressed_tensors patches from_pretrained. load_context installs that patch and MoE linearization. Using load_quantizable_moe alone gives ValueError: ... but found auto_offload.

2. Both context managers patch AutoModelForCausalLM by default. For a multimodal architecture loaded via AutoModelForImageTextToText, pass the class explicitly or the patch lands where nothing calls it.

3. Never pass max_memory for a model larger than GPU+CPU. With a budget set, planning goes through infer_auto_device_map, whose plan for such a model is {"": "disk"} — which dispatch_model rejects: "You are trying to offload the whole model to the disk." Verified on the meta device: every budget from 40 to 100 GiB returned "1 module, on disk", including with no_split_module_classes=[].

4. init_dist() / torchrun belong to the DDP examples, not the disk-offload ones. Carrying them across produces the same all-disk refusal. Isolate with a controlled comparison: identical load call under plain python3 vs torchrun.

5. REAP pruning refuses to share a calibration pass. REAPPruningModifier must be the only modifier in the recipe during calibration. Fix is pipeline="independent".

6. sequential_targets will not split a layer below its decoder-layer class. Passing ["<DecoderLayerClass>", "Embedding"] to cut the graph finer did not repartition — still 49 subgraphs, identical OOM. Embedding is not accepted as a cut point.

The extra_cpu_mem reserve is the unified-memory knob

load_offloaded_model(model_class, extra_cpu_mem=5e9) — default reserve 5 GB, and the CPU budget is exactly psutil.virtual_memory().available - extra_cpu_mem.

On a discrete GPU, filling host RAM costs the GPU nothing. On unified memory it starves the GPU of the pool it needs for its own context. With available = 127.2 GB:

load_context() hardcodes 5e9 and does not forward it, so tuning means calling load_offloaded_model(cls, extra_cpu_mem=…) and load_quantizable_moe(cls) directly.

The band is narrow, and both bounds are set by that one layer.

How far the module route gets

Best run at reserve 12: dispatched across {disk, cpu}, REAP initialized (48 MoE layers, 512 experts each, dropping 128), quantization config applied to 98,869 modules, all 49 subgraphs traced, subgraphs 1 and 2 calibrated — then CUDA error: out of memory on subgraph 3, which is layer 1.

Everything works except the one layer.

What actually works: convert at the file level

compressed_tensors.entrypoints.convert.convert_checkpoint — "Convert a model checkpoint … without loading it up in memory, instead operating directly on the model safetensors files":

convert_checkpoint(model_stub, save_directory, converter, max_workers=2)

You supply a Converter with process(tensors) -> tensors, validate, get_dependencies, and create_config. At this level the n-gram shards are ~0.8 GB tensors, so the 102 GB module is never built — and no GPU is involved, for real this time.

To quantize a bare tensor, NVFP4PackedCompressor.compress(state_dict, scheme) takes {weight, weight_scale, weight_global_scale}. Computing the scales is the caller's job and the convention is undocumented; what works is a two-level scale — one fp32 global scale mapping the tensor's amax onto the product of the format maxima, then calculate_qparams per group of 16:

global_scale = (FP8_E4M3_DATA.max * FP4_E2M1_DATA.max) / w.abs().max()
grouped = w.reshape(w.shape[0], -1, 16)
scale, _ = calculate_qparams(grouped.min(-1).values, grouped.max(-1).values,
                             scheme.weights, global_scale=global_scale)

Verify before converting hundreds of GB — a wrong convention gives a wild error, not a plausible one. Round-tripping one real shard through unpack_fp4_from_uint8:

shape (2500012, 160)   0.800 GB bf16 -> 0.225 GB packed (3.56x)
mean abs err 5.31e-04   mean relative err 8.96%   (weight amax 5.47e-02)

8.96% is the honest cost of 4 bits on this table. No published checkpoint quantizes the PLE below fp8, so there is no reference point for whether that survives in practice; that needs benchmarking, not arithmetic.

130 shards × 0.225 GB = 29.3 GB, against 28.8 GB projected from the sizing formula.

Budget, if the converted checkpoint reloads

With layer 1 at ~29 GB instead of 102 GB, and 25% REAP expert pruning on a subsequent ordinary calibrated pass:

leaving ~25 GB for KV and activations.

Open question: whether a checkpoint carrying shard_N.weight_packed reloads, given the loader expects shard_N.weight to assemble into one Embedding. Untested at time of writing.

Bottom line

None of this is a bug in llm-compressor — its disk-offload path is built for hosts where CPU RAM is free real estate. Unified memory breaks that assumption, and this model concentrates 29% of its parameters into a single module, which breaks it hard.

If you have two accelerators, use TP2 and ignore all of the above.