LoRA: A parameter-efficient technique (PEFT) based on low-rank adapters. Instead of training all the parameters, we only train these adapters. QLoRA: Another PEFT based on LoRA, which also quantizes the weights
Training a causal language model from scratchby Hugging Face: Pre-train a GPT-2 model from scratch using the transformers library. TinyLlamaby Zhang et al.: Check this project to get a good understanding of how a Llama model is trained from scratch. ...