5 results

for transformers

• Mixture of experts: grow the parameters, not the compute — and pay in routing
  The sparsely-gated MoE layer (Google Brain, 2017 — published within months of the Transformer itself) uses feedforward networks as experts and a linear-softmax gate over the top-k scores, with noise added to the gate to help load balancing. Typical k is 1 or 2; k=1 is the Switch …
  field/mixture-of-experts · moe, routing, inference, transformers, llm
• Quantizing Qwen3.8-Flash-Next on one unified-memory box
  Field notes from fitting `Qwen/Qwen3.8-Flash-Next` (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB *unified* memory, aarch64, CUDA 13, sm_121. Measured with `llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `transformers` 5.16.1, `torch` 2.11.0+cu130…
  field/qwen38-flash-next-on-one-unified-memory-gpu · quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors
• Tokenizers are a compression table's cut points, not text
  What a language model consumes is a sequence of integers into a fixed vocabulary. The thing that turns text into those integers is a tokenizer, and for most current large language models it is a modif
  field/llm-tokenization · tokenization, llm, bpe, inference, transformers
• KV caching: the memory bill autoregressive decoding runs on
  An autoregressive transformer generates one token per step, and at every step the new token's query vector attends over the keys and values of every token before it. The query changes each step; the a
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• Speculative decoding: a cheap model guesses, an expensive one signs off in one pass
  Speculative decoding accelerates autoregressive generation by producing several tokens per decoding step instead of one — without changing what the big model would have said. A small draft model propo
  field/speculative-decoding · speculative-decoding, inference, llm, latency, transformers