lora_module_name ="q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj" HF上,LoraConfig类target_module的配置要根据具体的LLM模型源码中的关键字来设置,才能准确的对模型参数进行微调。 参考 推荐阅读LoRA理论及实战相关文章。
Owner hiyouga commented Jan 11, 2024 We used the default q_proj,v_proj modules just to estimate the minimum resource usage. You can use all linear layers with LoRA adapters to achieve a better fitting. 👍 1 hiyouga added the solved label Jan 11, 2024 gotzmann closed this as comple...