例如,如果您在同一 GPU 上运行两个 vLLM 实例,则可以为每个实例将 GPU 内存利用率设置为 0.5。 --guided-decoding-backend {outlines,lm-format-enforcer,xgrammar} 默认情况下,哪个引擎将用于引导解码(JSON 模式/正则表达式等)。目前支持 https://github.com/outlines-dev/outlines,
可通过请求中的guided_decoding_backend参数覆盖。 --distributed-executor-backend{ray,mp}用于分布式服务的后端。当使用多于1个GPU时,如果安装了"ray"将自动设置为"ray",否则设置为"mp"(多进程)。 --worker-use-ray 已弃用,请使用--distributed-executor-backend=ray。 --pipeline-parallel-size PIPELINE_PARALLEL...
当LLMEngine初始化时,会在_build_logits_processors()方法中调用get_local_guided_decoding_logits_processor()方法获取当前可用后端对应的LogitsProcessor(位于vllm/model_executor/guided_decoding目录下)。 此时,需要传入 Guided Decoding 相关的参数GuidedDecodingParams,这些参数位于SamplingParams中,可以在启动 vLLM 时进...
例如,如果您在同一 GPU 上运行两个 vLLM 实例,则可以为每个实例将 GPU 内存利用率设置为 0.5。 --guided-decoding-backend {outlines,lm-format-enforcer,xgrammar} 默认情况下,哪个引擎将用于引导解码(JSON 模式/正则表达式等)。目前支持 https:///outlines-dev/outlines, https:///mlc-ai/xgrammar, 和 http...
backend is None: backend_name = request.sampling_params.guided_decoding.backend_name if backend_name == "xgrammar": self.backend = XgrammarBackend(self.vllm_config) grammar = self.executor.submit(self._async_create_grammar, request) # 异步调用:self.backend.compile_grammar(...) request....
A high-throughput and memory-efficient inference and serving engine for LLMs - Revert "[Misc][Bugfix] Disable guided decoding for mistral tokenizer"…· vllm-project/vllm@02c9afa
async def test_guided_json_completion(server, client: openai.AsyncOpenAI, guided_decoding_backend: str): completion = await client.completions.create( model=MODEL_NAME, prompt=f"Give an example JSON for an employee profile " f"that fits this schema: {TEST_SCHEMA}", n=3, temperature=1.0, ...
[str] = Field( default=None, description=( "If specified, the output will follow the context free grammar."), ) guided_decoding_backend: Optional[str] = Field( default=None, description=( "If specified, will override the default guided decoding backend " "of the server for this specific ...
disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=qwen) INFO: Started server process [614] INFO: Waiting for...
bfloat16, max_seq_len=128, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend=...