make torch.compile work with vLLM (facebook/opt-125m , meta-llama/Llama-2-7b-hf, meta-llama/Llama-3-8b-hf) models #48209 Sign in to view logs Summary Jobs assign Run details Usage Workflow file Triggered via issue July 19, 2024 18:29 laithsakka commented on #130174 125be00 Sta...
llm = LLM(model="facebook/opt-125m") # Generate texts from the prompts. outputs = llm.generate(prompts) To use torch.compile, we need to add self.model = torch.compile(self.model) in this line: https://github.com/vllm-project/vllm/blob/main/vllm/worker/model_runner.py#L253 . ...
can metaseq 125M to 66B OPT model can be used as BB3 model just like 175B model work #4988 Closed MrD005 opened this issue Mar 23, 2023· 5 comments Comments MrD005 commented Mar 23, 2023 i am trying to change BB3 2.7B model with 6.6B OPT model using metaseq but it is not ...
Assign User on Comment make torch.compile work with vLLM (facebook/opt-125m , meta-llama/Llama-2-7b-hf, meta-llama/Llama-3-8b-hf) models #50556 Sign in to view logs Summary Jobs assign Run details Usage Workflow file Triggered via issue July 24, 2024 16:39 anijain2305 commented ...