然而,随着模型规模的不断增大,深度学习模型的训练变得耗时且昂贵,设计最优的并行策略组合以提高其在多设备上的执行性能是目前该领域的一大挑战。在此背景下,微软亚洲研究院提出 nnScaler 技术,通过一套并行化原语和策略限定搜索的方法来寻求最佳的并行策略组合。这一尝试为寻求深度学习并行策略最优化提供了方案和工具,...
To avoid space explosion, nnScaler allows the application of constraints to those primitives during space construction. With the proposed primitives and constraints, nnScaler can compose existing search spaces as well as new ones. Experiments show that nnScaler can ...
nnScaler is a parallelization engine that compiles a Deep neural network (DNN) model that designed for single-GPU execution into a program that capable of running in parallel across multiple GPUs. System Highlights: Ease of Use: Only a few lines of code need to be changed to enable automated...
【微软重塑深度学习并行策略】微软的研究人员们重新审视了并行策略的现有实现方式,并行化的过程实际上就是将原本针对单一设备执行的计算数据流图重新进行分割变换,从而变成针对多个设备执行的分布化数据流图。研究人员提出了一套基本操作作为并行化原语,包含 op-trans 表示如何对算子和张量进行划分,利用这套原语,数据流图...