Parameter-Efficient Fine-Tuning of Large Pretrained Models...
Given a pretrained model with a weight matrix 𝑊∈ℝ𝑑×𝑘W∈Rd×k, the conventional fine-tuning approach updates W to 𝑊+Δ𝑊W+ΔW. Instead, LoRA represents the update Δ𝑊ΔW as a product of two smaller matrices A and B: Δ𝑊=𝐵×𝐴ΔW=B×A (1) where 𝐵∈...