针对你遇到的问题“auto_gptq.nn_modules.qlinear.qlinear_cuda extension not installed”,这里有几个可能的解决方案,你可以按照以下步骤逐一尝试: 检查CUDA是否正确安装并配置: 你可以通过运行以下命令来检查CUDA是否已正确安装并配置: bash nvcc --version 或者 bash nvidia-smi 如果这两个命令能够正常显示CUD...
值得注意的是,这个问题很有可能是因为安装不正确,所以 GPTQ 无法正确使用 GPU 进行推理,也就是说无法进行加速,即便 print(model.device) 显示为 "cuda"。类似的问题见 Is This Inference Speed Slow? #130 和CUDA extension not installed #694。 这个问题是普遍存在的,当你直接使用 pip install auto-gptq 进...
while running code even though i have Cuda installed Exllama kernel is not installed, reset disable_exllama to True. This may because you installed auto_gptq using a pre-build wheel on Windows, in which exllama_kernels are not compiled. To use exllama_kernels to further speedup inference, ...
include dirs are: ['autogptq_cuda'] running bdist_wheel C:\Users\wuyux\anaconda3\envs\localgpt\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils back...
print(f"Building cuda extension requires PyTorch (>=1.13.0) being installed, please install PyTorch first: {e}") sys.exit(1) CUDA_VERSION = None ROCM_VERSION = os.environ.get('ROCM_VERSION', None) if ROCM_VERSION and not torch.version.hip: ...
nope:python convert-to-gptq.py -m ./gpt-cmd -o ./gpt-cmd-gptqCUDA extension not installed....
(from https://pypi.tuna.tsinghua.edu.cn/simple/torch/)你可能应该使用CUDA 12和现代版本的PyTorch,...
RUN /bin/bash -o pipefail -c 'cd /root/AutoGPTQ && PATH=/usr/local/cuda/bin:"$PATH" TORCH_CUDA_ARCH_LIST="8.0;8.6+PTX" BUILD_CUDA_EXT=1 python setup.py install' But it's still not building the kernel: logs: WARNING:CUDA extension not installed. WARNING:The safetensors archive ...
WARNING:auto_gptq.nn_modules.qlinear_old:CUDA extension not installed. Seems auto-gptq's CUDA extension not be installed properly, that's the reason why inference is slow for you. Maybe you should try to re-install auto-gptq and see any improvements Contributor TheBloke commented Jun 3, ...
Intel® Gaudi® 2 AI acceleratorBUILD_CUDA_EXT=0 pip install auto-gptq --no-build-isolation2.3.1+Intel Gaudi 1.17 AutoGPTQ can be installed with the Triton dependency withpip install auto-gptq[triton] --no-build-isolationin order to be able to use the Triton backend (currently only ...