首先将HF_ENDPOINT环境变量修改为 hf-mirror.com: export HF_ENDPOINT=https://hf-mirror.com 然后进行下载: huggingface-cli download --resume-download Qwen/Qwen2-72B-Instruct-GPTQ-Int4 --local-dir Qwen2-72B-Instruct-GPTQ-Int4 这会在当前目录中创建一个Qwen2-72B-Instruct-GPTQ-Int4目录用于保存模...
运行命令: python -m vllm.entrypoints.openai.api_server --served-model-name Qwen2-VL-7B-Instruct --model /data1/MLLM/qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 --tensor-parallel-size 8 --gpu-memory-utilization 0.8 --cpu-offload-gb 10 --port 5001 --host 0.0.0.0 --quantization gptq Value...
[Bug]: vllm-0.5.3.post1部署Qwen2-72b-instruct-awq模型，刚开始服务正常，但是并发高的时候就...
想问下我们目前是否支持部署,如果不能部署的话预计什么时候可以支持一下~Collaborator Jintao-Huang commented Aug 5, 2024 已经支持了 文档有写～ Jintao-Huang closed this as completed Aug 8, 2024 Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment ...