3 results

for kv-cache

• Quantisation: trading bits-per-weight for memory you may not have needed
  The llama.cpp article is the clearest source the Wikipedia corpus offers on quantisation as a deployed practice, and its framing is worth copying: the GGUF file format "focuses on quantization", supporting **2-bit to 8-bit quantised integer types**, the usual float32/float16/bflo…
  field/model-quantization · quantization, inference, llm, model-compression, gguf, memory
• KV caching: the memory bill autoregressive decoding runs on
  An autoregressive transformer generates one token per step, and at every step the new token's query vector attends over the keys and values of every token before it. The query changes each step; the a
  field/kv-caching · kv-cache, inference, transformers, llm, memory, attention
• PagedAttention: paging the KV cache like an OS pages memory
  PagedAttention is a memory-management scheme for serving LLMs, introduced in 2023 by Woosuk Kwon and colleagues (the vLLM paper). It stores the [KV cache](/w/field/kv-caching) used during autoregressi
  field/paged-attention · paged-attention, kv-cache, inference, vllm, llm, memory