DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepSpeed/deepspeed at master · Marcus-Arcadius/DeepSpeed
master .github DeepSpeedExamples azure bin csrc deepspeed docker docs op_builder release requirements tests benchmarks model onebit perf small_model_debugging unit conftest.py .clang-format .gitignore .gitmodules .pre-commit-config.yaml .pylintrc ...
deepseed新发布了一个训练方式autoTP,deepseed就是玩zero的,大家都知道,zero是一种配合DP省显存的方式,zero1是优化器被打散,zero2是梯度,zero3直接是模型参数,如果手里卡有限,显存不够,一般是采用FSDP+zero3来训练才能装下比较大的模型,但是zero3因为能拆的都拆了,所以一通信就all2all,计算全被通信吃了,MFU就...
This branch is 259 commits behind microsoft/DeepSpeed:master. Contribute Latest commit Git stats History Files Failed to load latest commit information. Type Name Latest commit message Commit time . . README.md attach.sh azure_config.json azure_ssh.sh build_docker_image.sh ...
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. - DeepSpeed/bin at master · finetunej/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepSpeed/docs/assets at master · jiang-gy/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepSpeed/.clang-format at master · traincheck-team/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepSpeed/blogs at master · microsoft/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. - DeepSpeed/Dockerfile at master · jl-massey/DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. - DeepSpeed/scripts at master · microsoft/DeepSpeed