NCCL_DEBUG=INFO should also tell you that. In terms of the number of GPUs supported, I am not aware of the limitation of only supporting up to 8 GPUs when NVSwitch/NVLinks are not present, at least in recent versions of NCCL. Previously there was indeed error like "peer mapping ...
[pip3] nvidia-nccl-cu12==2.21.5 [pip3] nvidia-nvjitlink-cu12==12.6.85 [pip3] nvidia-nvtx-cu12==12.6.77 [pip3] onnx==1.16.2 [pip3] onnxruntime==1.20.1 [pip3] onnxruntime_extensions==0.13.0 [pip3] onnxscript==0.2.0 [pip3] pytorch-triton==3.0.0+45fff310c8 [pip3]...
dim=1)defremove_fast_pass(x_with_fastpass:torch.Tensor):returnx_with_fastpass.clone()[:, :-1, :]defget_fast_pass(x_with_fastpass:torch.Tensor):returnx_with_fastpass.clone()[:,-1, :]defflip_fast_pass_token(x_with_fastpass:torch.Tensor):output=x_with_fastpass.clone()output...
(nonzero,), self._out_spec) # To see more debug info, please use `graph_module.print_readable()` E1112 16:59:49.656000 784 torch/_subclasses/fake_tensor.py:2017] [2/0] failed while attempting to run meta for aten.sym_constrain_range_for_size.default E1112 16:59:49.656000 784 ...
[pip3] nvidia-nccl-cu12==2.21.5 [pip3] nvidia-nvjitlink-cu12==12.4.127 [pip3] nvidia-nvtx-cu12==12.4.127 [pip3] onnx==1.16.1 [pip3] onnxconverter-common==1.16.0 [pip3] onnxmltools==1.13.0 [pip3] onnxruntime==1.19.0 [pip3] onnxruntime_extensions==0.13.0 [pip3] ...
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi [conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi [conda] torch 2.5.1 pypi_0 pypi [conda] torchvision 0.20.1 pypi_0 pypi [conda]...
MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=...
If no one has time to actually recreate this and debug it, please just tell me if full QAT to ONNX support is planned or not (I've seen contributors saying it is not planned, I've seen posts where there is said that it should be in torch 2.0, so maybe just a clear answer would...
We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Cancel Create saved search Sign in Sign up Reseting focus {...
Search or jump to... Search code, repositories, users, issues, pull requests... Provide feedback We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your...