bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 ) model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=bnb_config) Expected behavior when i am running the above snipped it is...
Please open individual issues if you still experience this and be sure to try with the newest version of bitsandbytes (pip install --upgrade bitsandbytes) before reporting. Then, if reporting, please consider what is the minimal code example and library versions that reproduce this error and b...
use_qlora: print(f"Using QLoRA - {torch_dtype}") quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch_dtype, bnb_4bit_quant_storage=quant_storage_dtype, ) els...
Could not find the bitsandbytes CUDA binary at PosixPath('/public/home/sb/anaconda3/envs/ft/lib/python3.10/site-packages/bitsandbytes-0.44.0.dev0-py3.10-linux-x86_64.egg/bitsandbytes/libbitsandbytes_cuda121.so') Could not load bitsandbytes native library: /public/home/sb/anaconda3/envs...
importlib._bootstrap>", line 241, in _call_with_frames_removed File "/usr/local/lib/python3.10/dist-packages/transformers/integrations/bitsandbytes.py", line 21, in <module> import bitsandbytes as bnb File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/__init__.py", line 15, ...
import os import bitsandbytes as bnb import torch import gradio as gr from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig access_token = os.environ["GATED_ACCESS_TOKEN"] quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb...