Triggered via pull request March 14, 2025 15:45 gaugarg-nv opened #12394 gaugarg-nv:enable_cuda_graph_on_11.x Status Action required Total duration – Artifacts – This workflow is awaiting approval from a maintainer in #12394 server.yml on: pull_request server-windows Matrix: server Waiting for pendi...
A high-throughput and memory-efficient inference and serving engine for LLMs - Enable CUDA graph for GPTQ & SqueezeLLM (#2318) · tjlcast/vllm@6ef00b0
[CUDA] Enable CUDA Graph on CUDA Toolkit < 12.x #9047 Sign in to view logs Summary Jobs labeler Run details Usage Workflow file labeler succeeded Mar 14, 2025 in 8s Set up job 1s Run actions/checkout@v4 2s Run actions/labeler@v5 2s Post Run actions/checkout@v4 0s ...
Currently, when performing GPU decoding/encoding, filter graph is disable. We should enable filter graph so that it is more flexible for GPU processing. implementation direction Explicitly pass HW device context and/or HW frame context t...
🚀 The feature, motivation and pitch vLLM only enables cuda graph for decoding-only batches (mainly because it didn't see big perf improvement if batched token length > 256). This behavior is preserved even after chunked prefill is enable...
BoyuanFeng:bf/cudagraph-enable-input-mutation-support Status Failure Total duration 3h 23m 3s Artifacts – pull.yml on: pull_request linux-jammy-py3.8-gcc11 / build 15m 16s linux-focal-cpu-py3.10-gcc9-bazel-test / filter 14s linux-focal-cuda11.8-py3.10-gcc9 / build 24m 46s...
🐛 Describe the bug import torch @torch.compile(fullgraph=True) def f(q, k, v): q = torch.cos(q) with torch.backends.cuda.sdp_kernel(enable_flash=True): return F.scaled_dot_product_attention(q, k, v) f(*[torch.randn(1,8,1024,64, dtype=tor...
Tensors and Dynamic neural networks in Python with strong GPU acceleration - `torch.backends.cuda.sdp_kernel(enable_flash=True)` causes graph breaks · pytorch/pytorch@ec660c3
[CUDA] Enable CUDA Graph on CUDA Toolkit < 12.x #9047 Sign in to view logs Summary Jobs labeler Run details Usage Workflow file Triggered via pull request March 14, 2025 15:45 gaugarg-nv opened #12394 Status Success Total duration 15s ...