[Doc] Change distributed_executor_backend to mp (#287) Browse files ### What this PR does / why we need it? Fix `ValueError: Unrecognized distributed executor backend tp. Supported values are 'ray', 'mp' 'uni',
distributed_executor_backend="ray", 107+ enforce_eager=True,# reduce test time 108+ ) 109+ engine=LLMEngine.from_engine_args(engine_args) 110+ assertengine.model_executor.uses_ray vllm/config.py +3 Original file line numberDiff line numberDiff line change ...
invllm.EngineArgs, the argument should bedistributed_executor_backendinstead ofdistributed-executor-backend. 👋 Hi! Thank you for contributing to the vLLM project. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only runfastcheckCI which starts running only a ...
distributed_executor_backend=distributed_executor_backend, ) # Add the requests to the engine. @@ -229,8 +231,9 @@ def main(args: argparse.Namespace): args.max_model_len, args.enforce_eager, args.kv_cache_dtype, args.quantization_param_path, args.device, args.enable_prefix_caching, args...
found by the RLHF community, see OpenRLHF/OpenRLHF#721 . even when the world size is 1, we should respect the distributed_executor_backend specified from users.
A high-throughput and memory-efficient inference and serving engine for LLMs - Dockerfile.ubi: use distributed-executor-backend=mp as default · vllm-project/vllm@6f1bd87
ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server", "--distributed-executor-backend=mp"] ENTRYPOINT ["python3", "-m", "vllm.entrypoints.openai.api_server"] FROM vllm-openai as vllm-grpc-adapter Expand All@@ -193,4 +193,4 @@ RUN --mount=type=cache,target=/root...
Merged dtrifiromerged 1 commit intoopendatahub-io:mainfromdtrifiro:entrypoint-cleanup Jul 10, 2024 +2−2 dtrifirocommentedJul 10, 2024 this is the default when--worker-use-rayis not provided and more than 1 GPU is available Dockerfile.ubi: get rid of --distributed-executor-backend=mp ...