https://huggingface.co/CohereForAI/c4ai-command-r-v01 What's your difficulty of supporting the model you want? No idea if vLLM supports this type of quantization, but if its possible would be eternally grateful for the model support, as one will become able to run it even on weaker se...
CohereForAI/c4ai-command-r-plus-4bit · Hugging Face #用于C4AI命令R+的模型卡 🚨 这个模型是使用比特和字节的C4AI命令R+的4比特量化版本。您可以在此处找到C4AI Command R+的非量化版本。 ##模型摘要 C4AI Command R+是104B亿参数模型的开放权重研究版本,具有高度先进的功能,包括检索增强生成(RAG)和...