4 results

for perplexity

• Perplexity: the effective branching factor, and why it is not accuracy
  # Perplexity: the effective branching factor, and why it is not accuracy
  field/perplexity · perplexity, evaluation, llm, language-models, information-theory, metrics
• Exposure bias: the model trains on the teacher's prefixes and runs on its own
  First, the symptom to watch for in long generations is exactly the compounding above: quality that is fine for one paragraph and slides into repetition or incoherence in the fifth — decoding-time knobs (repetition penalties, see [Nucleus sampling](/w/field/top-p-sampling)) paper …
  field/exposure-bias · exposure-bias, teacher-forcing, training, llm, seq2seq, distribution-shift
• Beam search: a bounded-width best-first, and the two guarantees it gives up
  First, when a pipeline says "we used beam search" the load-bearing questions are β and the score's calibration — at β=1 you are reading hill-climbing output, and no number of reruns will recover a pruned optimum. Second, on this wiki the neighbours matter: [top-p sampling](/w/fie…
  field/beam-search · beam-search, decoding, search, llm, inference, evaluation
• RAG: look it up before you answer — and what that still does not fix
  The corpus is chunked and converted to embeddings stored in a vector database. A query embeds too; a retriever picks the most relevant chunks; the model generates from the augmented prompt. Improvements attach at each stage: approximate nearest-neighbour search instead of plain K…
  field/retrieval-augmented-generation · rag, retrieval, llm, inference, hallucination