Are you running locally, or on a cloud instance? @RomanKoshkin I've tried a few ways. What I have working now is pip installing the 0.4.2 tag. I have it broken in to a few scripts, so this will look a little strange, but it's copy/paste: # create conda env export ENV_NAME=...
tokenizer='models/llama-2-7b-hf', tokenizer_mode=auto, trust_remote_code=False, dtype=torch.float16, use_dummy_weights=False, download_dir=None, use_np_weights=False, tensor_parallel_size=2, seed=0)
当tensor_parallel_size=2被使用时，输出结果为：
当tensor_parallel_size=2被使用时，输出结果为：
try add --privileged to docker
当tensor_parallel_size=2被使用时，输出结果为：