{ "model": "my-model", "prompt": "My prompt", "top_p": 1, "n": 1, "frequency_penalty": 1.0, "stream": false, "max_tokens": 15, "nvext": { "guided_decoding_backend": "my-custom-backend" } } Custom guided decoding backend specifications To launch the custom guided decoding ...
NIM_GUIDED_DECODING_BACKEND=/name/of/subdirectory/in/NIM_CUSTOM_GUIDED_DECODING_BACKENDSRequests will useNIM_GUIDED_DECODING_BACKENDby default. To launch the container, use the following command: dockerrun-it--rm--name=$CONTAINER_NAME\--runtime=nvidia\--gpusall\--shm-size=16GB\-eNGC_API_KEY...