allgather_partitions:在每个step结束时,选择用allgather集合通信操作还是一系列的broadcast从所有gpus收集更新后的参数,一般不需要修改 overlap_comm:推荐设置为True,deepspeed将在梯度计算时尝试并执行梯度通信。可以有效的减少通信时间,从而加速整个训练过程。 reduce_scatter:默认为True,使用reduce_scatter来代替allreduce来...
overlap_comm使用的是allgather_bucket_size和reduce_bucket_size值的4.5倍。如果它们被设置为5e8,需要9GB显存(5e8 x 2Bytes x 2 x 4.5)。如果内存大小是8GB或更小,需要将这些参数减少到约2e8,从而避免OOM,这需要3.6GB显存。如果在大容量GPU上也出现OOM,也需要做同样的调整。 在deepspeed==0.4.4中新增了rou...
/home/user/code/目录下新建ds_config.json文件,写入: {"train_batch_size":"auto","train_micro_batch_size_per_gpu":"auto","gradient_accumulation_steps":"auto","gradient_clipping":"auto","zero_allow_untested_optimizer":true,"fp16":{"enabled":"auto","loss_scale":0,"initial_scale_power":...
"overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1e9, "reduce_bucket_size": 1e6, "stage3_prefetch_bucket_size": 0.94e6, "stage3_param_persistence_threshold": 1e4, "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_gather_16bit_w...
{"stage": 2,"allgather_partitions": True,"allgather_bucket_size": 2e8,"overlap_comm": True,"reduce_scatter": True,"reduce_bucket_size": 2e8,"contiguous_gradients": True,"cpu_offload": False}}model_engine, optimizer, _, _ = deepspeed.initialize(args=params,model=model,model_parameters...
"overlap_comm": true, "contiguous_gradients": true, "cpu_offload": true, "cpu_offload_params": false, "cpu_offload_use_pin_memory": false, "sub_group_size": 1e9, "stage3_prefetch_bucket_size": 5e7, "stage3_param_persistence_threshold": 1e6, ...
{ "zero_optimization": { "stage": 2, "offload_optimizer": { "device": "cpu", } "contiguous_gradients": true, "overlap_comm": true } } 如上所述,除了将stage字段设置为2(启用ZeRO Stage 2,但Stage 1也可以),我们还需要将offload_optimizer设备设置为cpu以启用ZeRO-Offload优化。此外,我们可以...
{"stage":3,"overlap_comm":true,"contiguous_gradients":true,"sub_group_size":1e9,"reduce_bucket_size":"auto","stage3_prefetch_bucket_size":"auto","stage3_param_persistence_threshold":"auto","stage3_max_live_parameters":1e9,"stage3_max_reuse_distance":1e9,"stage3_gather_16bit_...
"overlap_comm": true, "contiguous_gradients": true, "sub_group_size": 1e9, "reduce_bucket_size": "auto", "stage3_prefetch_bucket_size": "auto", "stage3_param_persistence_threshold": "auto", "stage3_max_live_parameters": 1e9, ...
{ "fp16": { "enabled": "auto", "loss_scale": 0, "loss_scale_window": 100, "initial_scale_power": 16, "hysteresis": 2, "min_loss_scale": 1e-10 }, "zero_optimization": { "stage": 2, "allgather_partitions": true, "allgather_bucket_size": 1e8, "overlap_comm": true, ...