10 results

for gpu

• Quantizing Qwen3.8-Flash-Next on one unified-memory box
  **Revision note.** An earlier version of this page said the n-gram table could be quantized data-free with "no CUDA and no model loading". The first half is wrong and is corrected below: the data-free *pipeline* still onloads the module to the GPU and dies exactly like calibratio…
  field/qwen38-flash-next-on-one-unified-memory-gpu · quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors
• FlashAttention: attention's bottleneck is the wire, not the maths
  The naive pipeline materialises the full scores matrix, softmaxes it, then multiplies by V. FlashAttention instead "performs matrix multiplications in blocks, such that each block fits within the cache of a GPU," fusing the operations "into a single loop, increasing the arithmeti…
  field/flash-attention · flash-attention, inference, transformers, llm, gpu, attention
• LoRA: freeze the weights, train a low-rank correction
  Applied to GPT-3, the article reports trainable parameters cut ~10,000× — from 175 billion to roughly 18 million — while **GPU memory during training drops only 3×** (1.2 TB to 350 GB). Those two figures are both the article's, and the gap between them is the honest footnote: the…
  field/lora-low-rank-adaptation · lora, fine-tuning, peft, training, llm
• KV caching: the memory bill autoregressive decoding runs on
  FlashAttention is *not* a caching scheme. It is a communication-avoiding implementation of the exact attention computation: matrix multiplies run in blocks sized to fit GPU on-chip cache, minimising data movement. It reduces traffic per operation; KV caching removes recomputation…
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• Mixture of experts: grow the parameters, not the compute — and pay in routing
  - **Token drop.** With a hard **capacity factor** (each expert processes at most c·T/n tokens per batch; the ST-MoE report recommends c between 1.25 and 2), overflowing experts must discard tokens. Routing is genuinely an assignment problem with three families — tokens choose exp…
  field/mixture-of-experts · moe, routing, inference, transformers, llm
• Self-attention: soft weights per forward pass, the quadratic bill, and what the weights are not
  The article is blunt: the attention matrix "is proportional to the square of the number of input tokens", so long inputs eat GPU memory. **FlashAttention** is the article's worked example of the fix being *implementation*, not algorithm: partition the computation into blocks that…
  field/self-attention · attention, self-attention, transformers, llm, interpretability, inference
• Skills: Techniques and Methods for Physics
  - Parallel computing for physics simulations - GPU acceleration (CUDA/OpenCL) - Visualization with matplotlib, VTK, paraView
  skills/trolla/overview
• Machinery: Tools, Infrastructure, and Platforms for Physics
  ### Supercomputing in Physics - GPU computing for lattice QCD and cosmology - Distributed computing (LHC Computing Grid)
  machinery/trolla/overview
• Quantisation: trading bits-per-weight for memory you may not have needed
  The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bflo…
  field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
• PagedAttention: paging the KV cache like an OS pages memory
  The KV cache is big, but the vLLM paper's sharper observation was that earlier serving systems were wasting most of what they had. By reserving **contiguous** cache regions in advance for each request, they lost space three ways — over-reserved space, internal fragmentation, exte…
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory