bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 ) model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_confi
Could not find the bitsandbytes CUDA binary at PosixPath('/public/home/sb/anaconda3/envs/ft/lib/python3.10/site-packages/bitsandbytes-0.44.0.dev0-py3.10-linux-x86_64.egg/bitsandbytes/libbitsandbytes_cuda121.so') Could not load bitsandbytes native library: /public/home/sb/anaconda3/envs...