--lora_target_modules ALL \ --model_name 小黄 'Xiao Huang' \ --model_author 魔搭 ModelScope \ --deepspeed default-zero3# 单卡A10/3090可运行的例子 (Qwen2.5-7B-Instruct) # 显存占用:24GB CUDA_VISIBLE_DEVICES=0 swift sft \ --model_type qwen2_5-7b-instruct \ --model_id_or_path q...
--tuner_backend swift \ --dtype AUTO \ --output_dir output \ --dataset ms-bench \ --train_dataset_sample 5000 \ --num_train_epochs 2 \ --max_length 1024 \ --check_dataset_strategy warning \ --lora_rank 8 \ --lora_alpha 32 \ --lora_dropout_p 0.05 \ --lora_target_modules A...
--load_dataset_config true # merge-lora并使用vLLM进行推理加速 CUDA_VISIBLE_DEVICES=0,1 swift export \ --ckpt_dir output/qwen2-vl-72b-instruct/vx-xxx/checkpoint-xxx \ --merge_lora true CUDA_VISIBLE_DEVICES=0,1,2,3 swift infer \ --ckpt_dir output/qwen2-vl-72b-instruct/vx-xxx/checkp...
微调脚本: LoRA # https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_32b_chat/lora_mp/sft.sh# Experimental environment: A100# 2*49GB GPU memoryCUDA_VISIBLE_DEVICES=0,1 \swift sft \--model_type qwen1half-32b-chat \--sft_type lora \--tuner_backend swif...
config=LoraConfig(task_type=TaskType.CAUSAL_LM,target_modules=["q_proj","k_proj","v_proj","...
新建examples/train_lora/qwen2.5_7b_lora_sft.yaml 微调配置文件,微调配置文件如下: ### model model_name_or_path: xxx/xxx # 预训练模型路径 ### method stage: sft do_train: true finetuning_type: lora lora_target: all ### dataset
--lora_target q_proj,v_proj \ --output_dir ${output_dir} \ --per_device_train_batch_size...
lora_rank:微调中的秩大小。秩的值并不是越大越好,此处设置的8是LoRA原论文中测试的最优解,根据论文中的结果,1或者2这种很小的秩的表现也是很好的。lora_alpha:LoRA 微调中的缩放系数。lora_dropout_p:LoRA 微调中的 Dropout 系数。lora_target_modules:指定lora模块, 默认为['DEFAULT']. 如果lora_target_...
Chat/lora/train_2024-06-03-22-15 \--fp16 True \--lora_rank 8 \--lora_alpha 16 \--lora_dropout 0 \--lora_target q_proj,v_proj \--val_size 0.1 \--evaluation_strategy steps \--eval_steps 1000 \--per_device_eval_batch_size 2 \--load_best_model_at_end True \--plot_loss ...
lora_target: all deepspeed: examples/deepspeed/ds_z3_config.json dataset dataset: sft_data_test template: qwen cutoff_len: 1024 max_samples: 1000 overwrite_cache: true preprocessing_num_workers: 16 output output_dir: /LLM/results/2.0version-Qwen-14b-lora-adapt-continue-pinpai-3eps ...