In this work, we first introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed LowRank Adaptation (DoRA). DoRA decomposes the pre-trained weight into...
本文怀疑LoRA的这种限制可能源于同时学习量级和方向适应的挑战,这对LoRA来说可能过于复杂。本文提出LoRA的一种变体,表现出更类似于FT的学习模式,可以提高LoRA的学习能力。基于权重分解分析的发现,文章提出了DoRA(Weight-Decomposed Low-Rank Adaptation)。DoRA的目标是通过模仿FT的学习模式来提高LoRA的学习能力,同时保持参...
DoRA: Weight-Decomposed Low-Rank Adaptation 单位:NVIDIA, 香港科技大学 论文:arxiv.org/abs/2402.0935 CVPR 2023 论文和开源项目合集请戳—> github.com/amusi/CVPR20 ICCV 2023 论文和开源项目合集请戳—> github.com/amusi/ICCV20 在广泛使用的参数高效微调(PEFT)方法中,LoRA 及其变体由于避免了额外的推理...
转载自https://icml.cc/virtual/2024/oral/35576[字幕由openai/whisper-large-v3-turbo + Qwen/Qwen2.5-72B-Instruct-AWQ生成(zero-shot)]Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., , 视频播放量 618、弹幕量 0、点赞数 18、投硬
[7] DoRA: Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353. [8]: Delta-...
[7] DoRA: Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353.
[7] DoRA: Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353. [8]: Delta-LoRA: Zi, B., Qi, X., Wang, L., Wang, J., Wong, ...
[7] DoRA: Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353. [8]: Delta-LoRA: Zi, B., Qi, X., Wang, L., Wang, J., Wong, ...
[7] DoRA: Liu, S. Y., Wang, C. Y., Yin, H., Molchanov, P., Wang, Y. C. F., Cheng, K. T., & Chen, M. H. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv preprint arXiv:2402.09353. [8]: Delta-LoRA: Zi, B., Qi, X., Wang, L., Wang, J., Wong, ...
Last week, researchers proposedDoRA: Weight-Decomposed Low-Rank Adaptation, a new alternative to LoRA, which may outperform LoRA by a large margin. DoRA is a promising alternative to standard LoRA (annotated figure from the DoRA paper: https://arxiv.org/abs/2402.09353) ...