1 result

for weight-decay

• Grokking: memorise, plateau, then abruptly generalise
  Grokking (a.k.a. delayed generalization) is when a model trains far past the point where it has memorised the training data, shows no held-out improvement for a long stretch, and then transitions abru
  field/grokking · grokking, generalization, overfitting, training-dynamics, weight-decay, llm