Fusion torch.compile passes¶

vLLM applies a set of kernel/operator fusions at compile time (via custom torch.compile Inductor passes) to separate optimizations from model definitions and avoid breaking layer abstractions in model code. These fusions are controlled by fields in PassConfig and are automatically enabled at appropriate optimization levels.

Quick Reference¶

The table below maps each fusion to its controlling flag/config knob, the operations it fuses, what level enables it by default, and an indicative speedup. The Fullgraph column indicates whether the fusion requires the entire model graph to be visible (either via Inductor partition or splitting_ops=[]), and the last column indicates whether the fusion activates for all num_tokens or just on the low or high end.

Support Matrix¶

The table below lists the quantization schemes supported by each fusion on each platform. — means the fusion is not available on that platform. The latest and in-progress work is available in the tracking issue: #36066

* fuse_attn_quant support depends on the attention backend in use; not all backends support fused quantization output. See the fuse_attn_quant section for per-backend details.

† enable_sp and fuse_gemm_comms are only autoconfigured for SM90 today; other architectures support requires setting PassConfig.sp_min_token_num explicitly. SM100 support also requires setting VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel.

Enabling / Disabling Fusions¶

Fusions are exposed through PassConfig, which is nested inside CompilationConfig:

from vllm import LLM
from vllm.config import CompilationConfig, PassConfig

llm = LLM(
    model="...",
    optimization_level=2, # Default optimization level
    compilation_config=CompilationConfig(
        pass_config=PassConfig(
            fuse_norm_quant=True,
            fuse_act_quant=True,
            fuse_allreduce_rms=False,  # disable a specific fusion
        )
    ),
)

Fusions can also be enabled using command-line flags with any vllm ... command:

# Enable O2 defaults, but turn off allreduce fusion
vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False

# The above is equivalent to the more verbose:
vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

# Same syntax in other commands, e.g. vllm bench:
vllm bench latency --model=meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False

Fields set explicitly by the user always take precedence over optimization-level defaults.

Fusion Details¶

AllReduce + RMSNorm (fuse_allreduce_rms)¶

What it fuses. Fuses the tensor-parallel all-reduce collective with the subsequent residual add, RMSNorm, and optionally a quantization step into a single FlashInfer / TRT-LLM communication kernel. This fusion is only profitable for small num_tokens, so the fusion is only performed in the lower compiled range.

Patterns covered:

• AllReduce → RMSNorm(+residual_add): CUDA sm90+ with FlashInfer
• AllReduce → RMSNorm(+residual_add) → FP8 static quant: CUDA sm90+ with FlashInfer
• AllReduce → RMSNorm(+residual_add) → NVFP4 dynamic quant: CUDA sm100+ with FlashInfer

The maximum tensor size below which the fused kernel is used is hardware-dependent (64 MB for TP=2 on SM90/SM100) and configurable via PassConfig.fi_allreduce_fusion_max_size_mb.

Code locations.

• Pass: vllm/compilation/passes/fusion/allreduce_rms_fusion.py
• FlashInfer all-reduce: vllm/distributed/device_communicators/flashinfer_all_reduce.py
• Benchmark: benchmarks/kernels/benchmark_fused_collective.py

Attention + Quantization (fuse_attn_quant)¶

What it fuses. Fuses the attention output quantization directly after the attention computation, eliminating a full-precision memory round-trip of the attention output. Patterns covered:

Attention → FP8 static quant:

• TRITON_ATTN: CUDA, ROCm
• FLASHINFER: CUDA sm100+ with FlashInfer installed
• ROCM_ATTN: ROCm
• ROCM_AITER_UNIFIED_ATTN: ROCm with AITER

Attention → NVFP4 dynamic quant:

• FLASHINFER: CUDA sm100+ with FlashInfer installed

Other attention backends do not support fused output quantization yet.

Code locations.

• Pass: vllm/compilation/passes/fusion/attn_quant_fusion.py
• Attention backends: vllm/v1/attention/backends/

RoPE + KV-Cache Update (fuse_rope_kvcache)¶

What it fuses. Fuses the rotary positional embedding kernel with the KV-cache scatter/write into a single kernel, avoiding separate reads and writes of the key and value tensors.

Requires: AMD ROCm with AITER enabled, the rotary_embedding custom op active (automatic), and the kv_cache update op visible in the graph: either by using Inductor graph partition or removed from splitting_ops. If these conditions are set, the fusion is enabled automatically for optimization level O1 and above.

Code locations.

• Pass: vllm/compilation/passes/fusion/rope_kvcache_fusion.py

Sequence Parallelism (enable_sp)¶

What it fuses. Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather, splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs.

Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the AsyncTP pass (fuse_gemm_comms). SP is only applied above a minimum token threshold that is autoconfigured based on device capability and model hidden_size. Currently only active on H100/SM90 for models with hidden_size >= 8192. The threshold is configurable via PassConfig.sp_min_token_num.

The general transformation:

Input → AllReduce → RMSNorm → Output
becomes:
Input → ReduceScatter → local RMSNorm → AllGather → Output

Patterns covered:

• First block: AllReduce → RMSNorm → ReduceScatter → RMSNorm → AllGather
• Middle blocks: AllReduce → fused_add_RMSNorm → ReduceScatter → fused_add_RMSNorm → AllGather
• Both with optional → FP8 static quant suffix

Requires: use_inductor_graph_partition=True or piecewise compilation with static sizes divisible by tensor_parallel_size.

Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+.

Code locations.

• Pass: vllm/compilation/passes/fusion/sequence_parallelism.py

AsyncTP GEMM + Collective Overlap (fuse_gemm_comms)¶

What it fuses. After Sequence Parallelism transforms the graph, fuses GEMM kernels with the surrounding reduce-scatter (output projection) and all-gather (input projection) using torch.ops.symm_mem symmetric-memory primitives, overlapping communication and computation. This overlap is only profitable for large num_tokens, so the fusion (and preceding SP) is only performed in the higher compiled range above PassConfig.sp_min_token_num.

Patterns covered:

• GEMM → reduce-scatter → fused_matmul_reduce_scatter
• all-gather → GEMM → all_gather_matmul
• FP8 scaled variants of both patterns

Supported hardware: NVIDIA CUDA with symmetric-memory (torch.distributed._symmetric_memory) support.

On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled ( #27893)

VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ...

Code locations.

• Pass: vllm/compilation/passes/fusion/collective_fusion.py
• Sequence parallelism pass: vllm/compilation/passes/fusion/sequence_parallelism.py

QK Norm + RoPE (enable_qk_norm_rope_fusion)¶

What it fuses. Fuses the sequence: split QKV → reshape → Q/K RMSNorm → reshape → rotary embedding into a single fused_qk_norm_rope CUDA kernel.

# Unfused:
q, k, v = split(qkv)
q_norm = rms_norm(q.view(heads))
k_norm = rms_norm(k.view(kv_heads))
q_rope, k_rope = rotary_embedding(q_norm, k_norm, ...)

# Fused:
fused_qk_norm_rope(qkv, ...)

Supported hardware: CUDA (sm80+) only, tested only on sm90 and sm100.

Code locations.

• Pass: vllm/compilation/passes/fusion/qk_norm_rope_fusion.py
• CUDA kernel: csrc/ops.h (fused_qk_norm_rope)

RMSNorm + Quantization (fuse_norm_quant)¶

What it fuses. Combines the custom rms_norm / fused_add_rms_norm operations with subsequent quantization into a single fused kernel, eliminating an intermediate read/write of the full-precision activation tensor. Two variants are fused:

• Plain RMSNorm + quant: rms_norm(x) → quant_fp8(y)
• Fused-add RMSNorm + quant: fused_add_rms_norm(x, residual) → quant_fp8(y) — also updates the residual in-place.

Note that AITER fusions are currently in a separate pass in vllm.compilation.passes.fusion.rocm_aiter_fusion.

Supported quantization scheme/hardware combinations:

• FP8 static per-tensor: CUDA & HIP kernel
• FP8 dynamic per-token: CUDA & HIP kernel, AITER
• FP8 dynamic per-token-group (128/64): CUDA & HIP kernel, AITER

Code locations.

• Pass: vllm/compilation/passes/fusion/rms_quant_fusion.py
• ROCm AITER pass: vllm/compilation/passes/fusion/rocm_aiter_fusion.py
• CUDA/HIP kernels: csrc/layernorm_quant_kernels.cu

SiLU+Mul + Quantization (fuse_act_quant)¶

What it fuses. Fuses the silu_and_mul gate-up projection activation with subsequent quantization into a single kernel, avoiding materialization of the full-precision post-activation tensor.

Note that AITER fusions are in a separate pass in vllm.compilation.passes.fusion.rocm_aiter_fusion.

Supported quantization scheme/hardware combinations:

• FP8 static per-tensor: CUDA & HIP kernel
• NVFP4 dynamic: CUDA sm100+ only with FlashInfer
• FP8 per-token-group (128): ROCm AITER only

Code locations.

• Pass: vllm/compilation/passes/fusion/act_quant_fusion.py
• ROCm AITER pass: vllm/compilation/passes/fusion/rocm_aiter_fusion.py
• CUDA/HIP kernels: csrc/quantization/

RMSNorm + Padding (fuse_act_padding)¶

What it fuses. Fuses a residual add + RMSNorm with a subsequent padding operation that pads the hidden dimension to a multiple required by downstream AITER Triton GEMM kernels.

Requires: AMD ROCm with AITER RMSNorm enabled. Enabled by default in optimization level O1 and above when the hidden size is 2880 and AITER Triton GEMMs not enabled.

Code locations.

• Pass: vllm/compilation/passes/fusion/rocm_aiter_fusion.py (RocmAiterTritonAddRMSNormPadFusionPass)

See Also¶

• Optimization Levels — high-level presets that set fusion defaults.
• torch.compile in vLLM — how the Inductor pass pipeline works.
• Attention Backends — attention-specific kernel selection.