winglian deleted the drop-bnb-cuda-env branch December 4, 2024 17:25 winglian mentioned this pull request Dec 7, 2024 Should we use master branch or stable version? #2144 Open 3 tasks Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comme...
If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION`, for example, `make CUDA_VERSION=113`. The CUDA version for the compile might depend on your conda install, if using conda. Inspect CUDA version via `conda list | grep cuda`. To manually override the ...
Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090 Nvidia driver version: 565.77 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK ava...
INFO 02-07 05:08:11 __init__.py:190] Automatically detected platform cuda. INFO 02-07 05:08:13 api_server.py:840] vLLM API server version 0.7.2 � uINFO 02-07 05:08:13 api_server.py:841] args: Namespace(host=None, port=8000, uvicorn_log_level='info', allow_credentials=Fal...
I'm trying to follow this colab-notebook on my server (Ubuntu 22.04.5 LTS, Python 3.12.3, CUDA Version: 12.4) to fine-tune Qwen2 VL on my custom datasets. I got this problem OSError: Can't load the model for 'unsloth/qwen2-vl-7b-instruct-unsloth-bnb-4bit'. If you were tryin...
version of bitsandbytes: `pip install -U bitsandbytes`" ) # if not torch.cuda.is_available(): # import bitsandbytes as bnb # if not getattr(bnb, "is_multi_backend_refactor_preview", False): # raise RuntimeError( # "Current bitsandbytes only support cuda, please switch to multi_...
The results can vary depending on your model, input data, cuda version & what quant you are using & it is impossible to predict exact values. I have tried to take these into account & make sure the results are within 500MB. Below table I cross-check 3b,7b & 13b model memories given...
Your current environment PyTorch version: 2.1.1+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Microsoft Windows 11 Home GCC version: Could not collect Clang version: Could not collect CM...
RuntimeError: Input type (CUDABFloat16Type) and weight type (torch.cuda.HalfTensor) should be the same Then I changed it to pixel_values = pixel_values.to(llm_dtype).unsqueeze(0)(llm_dtype is llava models weight load dtype) RuntimeError: self and mat2 must have the same dtype, but...
from_dict(quant_state, device="cuda") # Second iterate over all prequant and normal weights # pre quantized weights would have a quant_state for weight_name, weight_tensor in self._hf_weight_iter( hf_weights_files, use_safetensors): # Filter out all weights whose suffix is not "....