Release Highlights

Editorial highlights to inform users about important upgrades to their models, hardware, and serving stack.

v0.7.5

Full changelog for v0.7.5 on GitHub →

FlashInfer v0.7.5 brings DeepSeek-V4.1 sparse MLA decode, an MXFP8-output batched GEMM and Qwen3.8 MoE shapes to Rubin, and extends the batch-invariant mode to attention, block-scale FP8 GEMM, BF16 MoE and sampling log-probabilities on Blackwell and Rubin. Blackwell serving adds a Kimi K3 MegaMoE, more MLA decode options and tree-structured speculative verification for Gated DeltaNet; Hopper, B200 and B300 get a W8A16 FP8 GEMM, and SM120 and SM121 gain modular NVFP4 fused MoE operators.

DeepSeek-V4.1 sparse MLA decode, MXFP8 projections and Qwen3.8 MoE shapes run on Rubin

The experimental Prims-TS decode kernel for DeepSeek-V4.1 sparse MLA (deepseek_v41_decode(backend="prims-ts")) runs on B200, B300 and Rubin SM107, reading the E4M3 sliding-window and packed FP4 compressed caches directly with CUDA Graph replay. bmm_mxfp8(backend="cute-dsl") on SM107 can write MXFP8 output with its block scales (dtype=torch.float8_e4m3fn plus out_scale), so DeepSeek-V4.1’s grouped attention output projection feeds the next MXFP8 GEMM in one launch. The Prims-TS FP8 block-scale routed MoE accepts 32x32 weight-scale blocks on SM100, SM103 and SM107, the format of the tensor-parallel shard of Qwen3.8-Flash-Next, and BatchPrefillWithRaggedKVCacheWrapper(backend="cute-dsl-rubin") adds opt-in E4M3 prefill with head dimensions 192 and 128 on SM107.

  • #6988 feat(deepseek_v41): experimental Prims-TS decode backend for DS4.1 sparse MLA (SM100/SM103/SM107)
  • #9003 feat(gemm): SM107 cute-dsl bmm_mxfp8 with MXFP8 output and strided A/out
  • #8666 feat(prims_ts): fp8 [32,32] block-scale routed MoE on Prims-TS
  • #9221 feat(prims_ts): enable DeepSeek fp8 32x32 scale blocks on SM107
  • #8716 Add opt-in Rubin CuTe DSL prefill backend

Batch-invariant mode reaches attention, block-scale FP8 GEMM, BF16 MoE and sampling log-probabilities

With batch_invariant=True, BatchDecodeWithPagedKVCacheWrapper, BatchPrefillWithPagedKVCacheWrapper and BatchPrefillWithRaggedKVCacheWrapper return output bits that depend only on a token’s KV position, head, window and input values, not on the batch, the page table or the SM count, with CUDA Graph replay (capture run() after plan()), on B200, B300 and Rubin, for BF16 causal attention with head dimension 64, 128 or 256. The same mode covers gemm_fp8_nt_groupwise(backend="auto", batch_invariant=True), softmax(batch_invariant=True), and sampling.log_softmax and sampling.log_probs (both with batch_invariant=True), and BatchInvariantBf16Config adds a batch-invariant BF16 backend to MoELayer, selected explicitly through BackendOptions.

  • #8590 feat(batch_invariant): batch-invariant attention behind `batch_invariant=True` on the paged decode, paged prefill and ragged prefill wrappers
  • #9280 feat(batch_invariant): `gemm_fp8_nt_groupwise(batch_invariant=True)`, the block-scale fp8 GEMM class
  • #9283 feat(batch_invariant): batch-invariant bf16 MoE backend for the unified `MoELayer` (`BatchInvariantBf16Config`)
  • #8999 feat(batch_invariant): `softmax(batch_invariant=True)`, `sampling.log_softmax` and `sampling.log_probs`: the row log-sum-exp class `lse.row`

Kimi K3 serving gains a MegaMoE and more MLA decode options on Blackwell

flashinfer.moe_ep adds a Kimi K3 and K3.1 Pro MegaMoE for the routed experts (E896, hidden 3584, intermediate 3072, top-16) through Sm100_KimiK3_MegaMoeConfig, at EP1, EP8 and EP16 with NVFP4 or MXFP4/MXFP8 precision and BF16 or MXFP8 combine; routing, latent projections, post-MoE RMSNorm and shared experts stay in the framework. Kimi K3 MLA decode (trtllm_batch_decode_with_kv_cache_mla(backend="cake")) adds a caller-owned LSE output and decode-context-parallel KV-shard masks on SM100 and SM103, and the flashinfer.mla.cake_kimi_k3_mla launchers add non-causal multi-token speculative verification.

  • #8934 feat(moe_ep): add isolated Kimi K3-family MegaMoE backend
  • #9067 feat(cake_kimi_k3_mla): add LSE, DCP and non-causal variants to the Cake Kimi-K3 MLA decode kernel

Gated DeltaNet verifies speculative-decode trees on Blackwell, and KDA decode takes fp32 state

gated_delta_rule_mtp(verify_parents=..., cache_replayssm=True, disable_state_update=True) verifies tree-structured drafts (EAGLE-style, more than one child per node) on SM100 and SM103 through the ReplaySSM tcgen05 kernel, with an FP32 state pool and 4 to 8 tokens per request, and gated_delta_rule_replayssm_commit(accept_paths=...) commits an accepted root-to-node path that is not a prefix. recurrent_kda’s one-warp and grouped CuTe decode kernels follow the state dtype, so BF16 and FP32 state pools both run, and CuTe KDA prefill with an explicit backend="cute-dsl" reads token-strided Q, K and V views of packed projections directly on SM100 and SM120.

  • #6155 feat(gdn): parent-indexed tree verify on the ReplaySSM tcgen05 kernel, with an fp64 reference
  • #6257 feat(gdn): accept_paths for tree commits; tracked commits on the tcgen05 fold
  • #8833 feat(kda): fp32 state for the CuTe recurrent decode kernels
  • #8613 feat: support token-strided QKV in CuTe KDA prefill

FP8-weight GEMMs with 16-bit activations run on Hopper and datacenter Blackwell

mm_bf16_fp8 and mm_fp16_fp8 multiply BF16 or FP16 activations by an FP8 E4M3 [N, K] weight with one FP32 scale on H100 (sm_90a), B200 and B300, dequantizing the weight inside the kernel. The kernels compile on first use through CuTe DSL and take caller-supplied output and workspace buffers.

  • #7495 feat(gemm): add mm_bf16_fp8 / mm_fp16_fp8, a W8A16 FP8 GEMM for Hopper and Blackwell (CuTe-DSL)

NVFP4 fused MoE gains modular operators on SM120 and SM121

On SM120 and SM121, the sm12x_nvfp4 NVFP4 backend of MoELayer adds modular operators (Triton routing, fused CuTe DSL FC1 activation with quantization, CuTe DSL FC2 finalization) beside the existing monolithic kernel, plus SiTU and non-default SwiGLU limits. On devices with 188 SMs, such as RTX PRO 6000, default SwiGLU chooses between them by shape and ReLU2 uses the modular operators; other devices keep the monolithic kernel. SiTU, non-default limits and explicit enable_pdl=True always use the modular operators.

  • #6510 feat(moe): add SM12x NVFP4 fused MoE

⚠️ Upgrade notices

  • The CAKE training kernels use one naming scheme: flashinfer.cake_<op>_{det,nondet}_train, with the mode in the name. Rename callers of the v0.7.4 names, such as flashinfer.cake_rmsnorm_train to flashinfer.cake_rmsnorm_det_train or flashinfer.cake_rmsnorm_nondet_train (default deterministic=True calls move to the det module and deterministic=False calls to nondet; context-parallel variants add _kcp); the old names have no aliases. The mapping is in docs/training_backend_migration.md, and the gated-activation MXFP8 quantizers are covered in docs/api/quantization.rst.

v0.7.4

Full changelog for v0.7.4 on GitHub →

FlashInfer v0.7.4 brings DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE to Rubin, and adds a batch-invariant mode for GEMMs and quantizers on Blackwell and Rubin that returns the same bits for a row at any batch size. Sparse-attention models get the SDSA indexer and a context-parallel MiniMax-M3 indexer, Gemma 4 decodes from an NVFP4 KV cache, and tensor-parallel layers fuse FP8 GEMM with all-reduce. Speculative-decode verification and sampling run faster on Hopper, and DeepSeek Sparse Attention indexing and Qwen3.5 speculative decode reach RTX PRO 6000.

DeepSeek-V4 sparse MLA decode and MXFP4 expert-parallel MoE run on Rubin

trtllm_batch_decode_sparse_mla_dsv4(backend="cake") runs its FP8 decode route on SM107 for DeepSeek-V4, and the experimental Prims-TS sparse MLA wrapper plans on Rubin, NVFP4 KV caches included. SM107 MegaMoE through MoEEpLayer adds MXFP4 activations with MXFP4 weights, kernel_variant="auto" selects between the inference and GenPhase variants through autotune_v2(), and the Lamport transport of NVLinkOneSidedAlltoAll for decode-sized steps builds for SM107. Rubin also gets a block-scaled contiguous grouped FP8 GEMM (prepare_group_gemm_fp8_nt_groupwise_contiguous), batch_deepgemm_fp8_nt_groupwise(backend="cake") for four weight shapes, the TinyGEMM backend of mm_bf16, and an SM107 kernel behind gated_delta_rule_mtp_ucache_flush for Gated DeltaNet MTP decode.

  • #8375 feat(cake_dsv4): SM107 runs the FP8 decode route of backend='cake'; the low-head prefill program takes PDL
  • #8186 perf(prims-ts): run sparse MLA on SM107 and trim the NVFP4 KV converter
  • #7944 feat(moe_ep): support MXFP4 W4A4 on SM107
  • #7911 feat(moe_ep): select Rubin MegaMoE variants with autotune v2
  • #8419 feat(moe_ep): build the NVLink one-sided Lamport transport for SM107
  • #8683 feat(cake_grouped_fp8_gemm): block-scaled contiguous grouped FP8 GEMM on SM107a
  • #8688 feat(cake_batch_deepgemm_fp8): masked BLOCK_N batch DeepGEMM FP8 on SM107a
  • #8227 fix(gemm): enable the TinyGEMM mm_bf16 backend on SM107
  • #6791 feat(gdn): Rubin (sm_107a) build of the UMMA ReplaySSM MTP u-cache flush kernel, with uGPU localization

Batch-invariant GEMMs and quantizers return the same bits for a row at any batch size

mm_bf16, bmm_fp8, mm_mxfp8 and mm_fp4 take batch_invariant=True on B200, B300 and Rubin: an output row’s bits depend only on the input values and the weight shape, independent of the batch size, the row’s position, the SM count and the configuration the dispatcher or autotuner picks. Reinforcement-learning pipelines can match the inference engine’s logits with the trainer’s bit for bit. mm_bf16 takes a bias in this mode, and mxfp8_quantize, mxfp4_quantize and nvfp4_quantize gain the same flag with numerics aligned to Transformer Engine. The flag requires backend="auto", and an unsupported input or device raises an error.

  • #8361 feat(batch_invariant): batch-invariant mode with `mm_bf16(batch_invariant=True)`
  • #8453 feat(batch_invariant): bmm_fp8(batch_invariant=True); the dispatcher becomes a ranking heuristic
  • #8535 feat(batch_invariant): bias for `mm_bf16(batch_invariant=True)`
  • #8536 feat(batch_invariant): `mm_mxfp8(batch_invariant=True)`, the MXFP8 GEMM class
  • #8537 feat(batch_invariant): `mm_fp4(batch_invariant=True)`, the NVFP4 and MXFP4 GEMM classes
  • #8565 feat(batch_invariant): batch-invariant quantizers: `mxfp8_quantize`, `mxfp4_quantize` and `nvfp4_quantize(batch_invariant=True)`

SDSA and MiniMax-M3 indexers select sparse-attention keys on Blackwell and Rubin

flashinfer.sdsa adds the indexer of SDSA (Simplified DSA), where a one-head indexer selects the Top-K keys of each query token for the token-sparse MQA decode: indexer_scores scores BF16 or NVFP4 paged keys for decode, quantize_write_index_qk_nvfp4 writes NVFP4 keys to the paged cache, and Top-K fusion (gather_key_sample, thresholds_from_sample, indexer_threshold_select, top_k_from_candidates) keeps only the keys above a per-row threshold for prefill. The APIs run on Blackwell SM100 and SM103, and indexer_scores with BF16 keys also runs on other GPUs such as H100. The experimental FP8MQATopKWrapper(backend="cutedsl") adds Selective TopK for single-request prefill with FP8, MXFP8 and MXFP4 queries and keys on B200 and B300. For MiniMax-M3, the experimental flashinfer.msa_ops index-context-parallel indexer covers TP2 and TP4 with page sizes 128 and 256 on B200, GB300 and Rubin, and dsa_indexer_det_topk(backend="cake") accepts FP8 queries and keys.

  • #8374 [SDSA] Add the SDSA (Simplified DSA) indexer for Blackwell, with BF16 scores on other GPUs
  • #7325 feat(attn_scores): add selective MQA TopK for prefill (indexer-throwaway topk)
  • #8122 feat(cake_msa_icp_indexer): MSA-compatible index-context-parallel indexer for TP2/TP4, P128/P256, SM100/SM103/SM107
  • #8749 feat(cake_dsa_indexer_det): opt-in FP8 (e4m3) form

Gemma 4 decodes from an NVFP4 KV cache on Blackwell, and tensor-parallel layers fuse FP8 GEMM with all-reduce

The experimental Prims-TS decode backend covers head dimension 512 with FP8 queries and an NVFP4 KV cache, the Gemma 4 attention geometry, with native BF16 output; Prims-TS attention also adds attention sinks, NHD caches, automatic runtime split-KV, and MXFP8 queries and keys for dense context attention on B200 and B300. On Blackwell, BatchPrefillWithRaggedKVCacheWrapper(backend="auto") serves sliding-window and attention-sink calls on cuDNN. For tensor-parallel serving, flashinfer.comm.fp8_rowwise_gemm_allreduce(backend="cake") runs a row-wise scaled FP8 GEMM and the all-reduce in one kernel at 2, 4 and 8 ranks on SM100 and SM103, and the CAKE NVFP4-transport all-reduce fuses the residual add and RMSNorm at four ranks for any row count. trtllm_fp4_block_scale_routed_moe(routing_metadata=...) runs the expert GEMMs from routing metadata the caller built earlier, for example on another stream.

  • #5720 feat(prim_ts): Add H512 FP8-Q/NVFP4-KV decode support
  • #6925 feat(prims-ts): broaden attention support and runtime split-KV
  • #7509 feat(prims-ts): QK-MXFP8 context FMHA
  • #5286 feat(prefill): teach the cuDNN ragged path sliding window, attention sinks and scalar scales
  • #8379 feat(cake_fp8_rowwise_gemm_allreduce): SM100 fused FP8 GEMM + all-reduce for TP2/4/8
  • #8605 feat(cake_fp8_rowwise_gemm_allreduce): SM103 fused FP8 GEMM + all-reduce for TP2/4/8
  • #8395 feat(cake_fp4_allreduce): four-rank fused all-reduce + residual + RMSNorm; any row count at four ranks
  • #8050 feat(moe): run the FP4 routed MoE expert body from caller-populated routing metadata

Mamba, Kimi K3 and Qwen hybrid models get packed causal-conv prefill, indexed state pools and a fused GDN decode block

flashinfer.mamba adds causal_conv1d_fn for packed variable-length prefill and causal_conv1d_update for decode and speculative verification on B200 and B300, with indexed state pools, fused SiLU and CUDA Graph replay. recurrent_kda(backend="small-bh") reads and updates an indexed state pool through ssm_state_indices for Kimi K3 prefill, and checkpointing_ssu accepts strided ReplaySSM trackers, so vLLM can keep them inside the per-slot Mamba state page. The experimental flashinfer.qwen_3_5_gdn_block runs the Gated DeltaNet sub-layer of Qwen 3.5-architecture models for one decode or MTP-verify step in a single call on B200, and the experimental Qwen 3.8 block adds extend and prefill entries for chunked prefill through the serving caches at any row count.

  • #8273 feat(mamba): add inference causal conv1d prefill and state update
  • #8287 feat(kda): indexed state pool (ssm_state_indices) for the small-BH prefill backend
  • #8548 feat(checkpointing_ssu): accept strided ring_start and prev_num_accepted
  • #8131 [Experimental] Qwen 3.5 GDN block: decode and MTP verify behind a backend registry, with an SM100 CuTe DSL backend
  • #7983 [Experimental] Qwen 3.8 Flash-Next block: the indexer's extend / prefill entries (chunked prefill through the serving caches) and the cuDNN Frontend 4c8e24355 re-vendor
  • #8058 [Experimental] Qwen 3.8 block: the extend / prefill entries take any row count (cuDNN Frontend 6b21be0e0 re-vendor)

Speculative-decode verification and sampling run faster on Hopper

On H100 and H200, BatchPrefillWithPagedKVCacheWrapper(backend="auto") serves batches whose requests carry at most 64 query tokens, such as MTP or speculative-decode verification, on FA2 with a query tile sized to the request. top_p_sampling_from_probs, top_k_sampling_from_probs and joint top_k_top_p_sampling_from_probs split each row across a thread-block cluster, which lowers sampling latency at decode batch sizes. The experimental qsa_sparse_paged_attention(backend="auto") runs faster for BF16, FP8 and NVFP4 caches on A100 and H100, and the SM90 BF16 MegaMoE backend adds opt-in compressed expert weights (wcx_max_tokens) for decode rounds.

  • #5188 prefill: use fa2 backend for small per-request query lengths
  • #5617 perf(sampling): split rows across thread block clusters in rejection samplers on SM90
  • #8423 perf(qsa): speed up BF16, FP8 and NVFP4 on SM80-SM90
  • #8089 feat(cake_mega_moe): SM90 BF16 opt-in compressed expert weights (WCX) for decode rounds

DeepSeek Sparse Attention indexing and Qwen3.5 speculative decode run on RTX PRO 6000

The experimental dsa_indexer_det_topk(backend="sm120") runs the DeepSeek Sparse Attention lightning indexer on SM120, scoring plus exact top-k in FP8 or MXFP4 with 32 or 64 heads, for DeepSeek-V3.2, DeepSeek-V4, DeepSeek-V4.1 and GLM-5.x in ragged prefill and paged decode forms. Gated DeltaNet ReplaySSM speculative decoding reaches SM120: gated_delta_rule_mtp(cache_replayssm=True) and gated_delta_rule_replayssm_commit run on RTX PRO 6000 in the Qwen3.5-397B geometry. BlockSparseAttentionWrapper(backend="vsa_sm120_blk64") runs on CAKE kernels.

  • #7085 feat(dsa_indexer): experimental SM120 backend for FP8 / MXFP4 prefill and decode
  • #7536 feat(gdn): add SM120 cuteDSL ReplaySSM GDN MTP decode kernels
  • #8799 perf(cake_vsa_sm120): run vsa_sm120_blk64 backend on Cake kernels

⚠️ Upgrade notices (Experimental)

  • To use APIs under flashinfer.experimental, install from source with pip install --no-binary flashinfer-python flashinfer-python==0.7.4 or pip install git+https://github.com/flashinfer-ai/flashinfer.git@v0.7.4, or use a nightly wheel from https://flashinfer.ai/whl.
  • Prims-TS attention: rename bmm1_scale and sm_scale to softmax_scale, bmm2_scale to output_scale, *_data_type to *_dtype, use_packed_q to packed_query, seq_lens to seq_lens_kv, max_seq_len to max_kv_len, and the MLA num_heads to num_qo_heads.

v0.7.3

Full changelog for v0.7.3 on GitHub →

FlashInfer v0.7.3 brings MTP and NVFP4 MLA decode to Rubin, and speeds up expert-parallel MoE decode steps on Blackwell while Hopper gains MXFP4 and NVFP4 experts. Hybrid Gated DeltaNet models gain paged recurrent state for prefix caching, DeepSeek Sparse Attention models get token-sparse decode, and MoELayer adds a low-latency small-batch path from Ampere to Blackwell.

DeepSeek-class MTP decode, Qwen3.5 deterministic MoE and Mamba2 prefill run on Rubin

BatchMLAPagedAttentionWrapper adds a cute-dsl-rubin-mtp backend for FP8 absorbed-MLA multi-token-prediction decode on SM107, and an experimental CAKE backend decodes dense MLA from an NVFP4 paged latent cache in the Kimi K3 and DeepSeek-V3 geometry. The deterministic unfused finalize of the CuTe DSL fused MoE works on Rubin for Qwen3.5 NVFP4 on Vera Rubin NVL72, and knobs="auto" tunes Rubin MegaMoE through autotune_v2(). Rubin also gets an SM107-native blockwise FP8 GEMM behind gemm_fp8_nt_groupwise(backend="auto"), Mamba2 prefill through SSDCombined(backend="cake"), and CAKE programs for DeepSeek routing, BGMV LoRA MoE, GDN and packed KDA decode.

  • #6177 feat(mla): add Rubin MTP backend to the planned wrapper
  • #6931 feat(cake_rubin_mla_nvfp4_decode): NVFP4 MLA paged decode for Rubin (sm_107a) from cake
  • #6950 feat(moe): unfused (deterministic) finalize path for the Rubin CuTe DSL fused MoE
  • #6535 feat(moe_ep): integrate Rubin automatic tuning with autotune v2
  • #6508 feat(gemm): SM107 CuTe-DSL blockwise FP8 GEMM backend for gemm_fp8_nt_groupwise
  • #6746 feat(cake_ssd_combined): Rubin sm_107a programs for the Mamba2 SSD combined prefill
  • #6238 feat(cake_kernel_rubin): Cake backends on Rubin (sm_107a / CC 10.7) — routing, BGMV MoE, concat-MLA-K, GDN decode/prefill, packed KDA T=1

Expert-parallel MoE decode steps run faster on Blackwell, and Hopper adds MXFP4 and NVFP4 experts

NVLinkOneSidedAlltoAll adds a Lamport transport for decode-sized steps on B200, B300, GB200 and GB300 (NVLinkOneSidedConfig(lamport=...)), which lowers dispatch and combine time for DeepSeek-V3, Kimi K3 and Qwen3 MoE decode steps, measured end to end on Kimi K3 in vLLM; the backend now shares its kernels with TensorRT-LLM and scales to 256 ranks. Blackwell MegaMoE adds an opt-in GenPhase decode pipeline, and knobs="auto" tunes the Blackwell backends through autotune_v2(). On Hopper, MoEEpLayer runs FP8-activation experts with MXFP4 or NVFP4 weights, and the new flashinfer.megamoe package adds fused NVFP4 and MXFP4 expert-parallel MoE on H100 and H200 for DeepSeek-V4-Flash-class models such as DeepSeek-V4-Flash-NVFP4.

  • #7498 feat(moe_ep): Lamport transport for small steps in NVLinkOneSidedAlltoAll
  • #6047 feat(moe_ep)!: sync NVLinkOneSidedAlltoAll with TensorRT-LLM's kernels; remove the deprecated MoeAlltoAll/MnnvlMoe
  • #6565 add-new-decoding-pipeline-for-blackwell-megamoe
  • #7523 feat(moe_ep): integrate Blackwell standard and GenPhase autotuning with v2
  • #4843 feat(moe_ep): add fused Hopper MXFP4 MegaMoE with autotuning
  • #6999 feat(moe_ep): add SM90 NVFP4 x FP8 CuTe DSL MegaMoE backend
  • #6343 [SM90] Add W4A8 MegaMoE support

MoELayer adds a low-latency small-batch path from Ampere to Blackwell and MXFP8 experts for Nemotron 3.5 Super

FrostLowLatencyConfig selects an opt-in cuDNN Frost backend of the unified MoELayer for routed MoE at one to eight tokens, covering BF16, FP16, FP8 (per-tensor and DeepSeek block), MXFP8 and NVFP4 experts and weight-only NVFP4, MXFP4, MXINT4 and E4M3 formats on SM80 (BF16 and FP16 only), SM90, SM100, SM103, SM107 and SM120. cuDNN grouped-GEMM backends (CudnnGroupedGemm*Config) join the same API as a further opt-in, and the cuDNN Frost grouped-GEMM backend covers BF16, MXFP8, NVFP4 and MXFP8-activation/MXFP4-weight experts on SM120, where auto-selection admits it. The CuTe DSL fused MoE adds MXFP8 activations with MXFP8 weights for Nemotron 3.5 Super’s non-gated ReLU² experts on B200 and B300, raising serving throughput at high concurrency over BF16 experts.

  • #6626 feat(moe): add unified Frost low-latency MoE backend
  • #7184 feat(moe): extend Frost low-latency to four-bit weight-only formats
  • #7162 feat(moe): add per-tensor FP8 and Hopper FMA to Frost
  • #7153 feat(moe): extend Frost low-latency FMA to FP16 and SM120
  • #5453 feat(moe): add cuDNN grouped-GEMM backends to the unified MoE API
  • #6504 Add SM120 Frost Moe Grouped Gemm Support for Swap-AB, MXFP8, NVFP4 and MXFP8×MXFP4
  • #6793 feat(moe): MXFP8 x MXFP8 mode for the CuTe-DSL fused MoE (for Nemotron models)

Hybrid Gated DeltaNet models serve with prefix caching, and Gated DeltaProduct gets decode and prefill kernels

flashinfer.gdn_paged keeps Gated DeltaNet recurrent state in a paged pool with a CSR page table, so a serving engine’s prefix cache can share and resume GDN state like paged KV: chunk_gated_delta_rule_paged writes a state at every page boundary and never touches prefix pages, gated_delta_rule_decode_paged updates pages per token, and gdn_prefix_materialize reconstructs a state at a fixed boundary from the decode checkpoint ring. GDN prefill also runs on Ampere. Gated DeltaProduct, the multi-step DeltaNet recurrence, gets cake_gated_delta_product_mtp for decode and MTP decode on H100, B200 and B300 and a backend="cake" chunked varlen prefill on H100. fused_kda_decode covers GLM-5.3-Flash’s 64 Kimi Delta Attention heads at TP1 to TP4 and a dimension-first conv-state layout for disaggregated serving.

  • #7785 feat(gdn): paged recurrent state for GDN prefill and decode
  • #5073 feat(gdn): prefix-cache state materialization kernel
  • #5109 feat(gdn): add SM8x prefill for the gated delta rule, fused and chunk-parallel
  • #6883 feat(cake_gdp): add Gated DeltaProduct decode backend (sm_100a, sm_103a)
  • #6942 feat(cake_gdp): Gated DeltaProduct decode on SM90 (sm_90a programs)
  • #6983 feat(cake_gdp): Gated DeltaProduct chunked varlen prefill on SM90 (sm_90a programs)
  • #7132 feat(kda): fused KDA decode for 16/64 heads and the DS conv-state layout

Token-sparse decode serves DeepSeek Sparse Attention models, and sparse MLA reads NVFP4 and FP8 caches

trtllm_batch_decode_with_kv_cache(sparse_top_k=K) runs token-sparse MQA decode on the trtllm-gen kernels, where each query token, MTP drafts included, attends only the keys its indexer selected; token_sparse_kv_indices builds the per-token page tables and group_union_masks lowers the same selection onto dense kernels. The experimental Prims-TS sparse-MLA decode wrapper serves DeepSeek Sparse Attention (DSA) decode for GLM-5.3 and DeepSeek-V3.2 from an E4M3 latent cache on B200 and B300, and adds NVFP4 and per-tensor FP8 cache recipes. Query sparse attention (QSA) gets cuDNN-backed qsa_forward and qsa_indexed_forward on B200, B300 and Rubin and an experimental qsa_sparse_paged_attention over NVFP4 paged caches on A100 and H100. MiniMax Sparse Attention ops (msa_*) for MiniMax M3 and MiniMax M3.1 Flash now run on H100.

  • #6420 [SDSA 1/6][Attention] Add token-sparse MQA decode to trtllm-gen
  • #6421 [SDSA 2/6][Attention] Add key unions and masks for token-sparse attention on dense kernels
  • #6536 feat(prims_ts): DSA (GLM-5.3 / DeepSeek-V3.2) sparse MLA decode with an E4M3 KV cache through the sparse MLA wrapper
  • #7410 feat(prims-ts): support NVFP4 and per-tensor FP8 sparse MLA caches
  • #7532 feat: add cuDNN 1.32 QSA forward APIs
  • #6804 feat(qsa_nvfp4): add NVFP4 QSA sparse paged attention
  • #7147 feat(qsa_sparse_paged_attention): backend="auto" default runs the Cake kernels on SM80 through SM90
  • #6140 SM90 MSA kernels for MiniMax M3

Sliding-window context attention reaches Blackwell and Rubin, and GLM-4.7 runs FP8 prefill on RTX PRO 6000

The TRTLLM-Gen context kernels accept a fixed sliding window and per-token VariableWindow bounds on B200, B300 and Rubin. backend="auto" routes more Blackwell shapes to cuDNN: single-token head-dim-256 paged decode in its measured CTA band, and causal BF16 paged prefill on B200 for both prefix-extension and full-sequence requests (cuDNN frontend 1.32 and cuDNN 9.27 or newer). FA2 takes a per-token, per-head FP8 KV scale for the dynamic fp8_per_token_head KV cache. On RTX PRO 6000, GLM-4.7 and GLM-4.7-Flash get FP8 prefill attention, and SM120 sparse MLA reads DeepSeek-V4.1 inline-scale KV records natively (kv_cache_format="fp8_dsv41_inline").

  • #4905 feat: integrate TRTLLM-Gen sliding-window FMHA
  • #6152 feat(cudnn): backend="auto" takes cuDNN for single-token d256 paged decode in the measured CTA band; CUDA-graph wrappers declare replay to cudnn-frontend 1.31+
  • #6231 [Attention] Reuse bounded cuDNN prefill plans and gate Blackwell paged auto
  • #7377 [Attention] Admit proven full paged prefill to Blackwell cuDNN auto
  • #5272 feat(attention): FA2 per-(token, head) FP8 KV scale
  • #6466 feat(attention): add SM12x FP8 prefill attention support for GLM-4.7 and GLM-4.7-Flash
  • #6616 feat(mla): read DeepSeek-V4.1 inline-scale KV records natively on SM120

⚠️ Upgrade notices

  • Install nvidia-cudnn-frontend>=1.32.0 and nvidia-cutlass-dsl>=4.8.0: pip install -U 'nvidia-cudnn-frontend>=1.32.0' 'nvidia-cutlass-dsl>=4.8.0'. Environments pinned to an older CuTe DSL must upgrade.
  • After upgrading flashinfer-python, run flashinfer download-kernels to replace mismatched flashinfer-cubin and flashinfer-jit-cache wheels; FlashInfer ignores wheels from another version and falls back to JIT compilation and on-demand cubin downloads.
  • Expert-parallel MoE: flashinfer.comm.MoeAlltoAll, MnnvlMoe and the flashinfer.comm.moe_a2a_* functions are removed together with the backend="trtllm" all-to-all kernels; use the flashinfer.moe_ep communication backends (NVLinkOneSidedAlltoAll, CakeAlltoAll, NVLinkTwoSidedAlltoAll, or SplitConfig(comm=...)), and rename dispatch(max_tokens_per_rank=...) to runtime_max_tokens_per_rank=....

⚠️ Upgrade notices (Experimental)

  • To use APIs under flashinfer.experimental, install from source with pip install --no-binary flashinfer-python flashinfer-python==0.7.3 or pip install git+https://github.com/flashinfer-ai/flashinfer.git@v0.7.3. The PyPI wheel carries every stable API and its JIT sources; the experimental kernel sources ship with the source install.

v0.7.2

Full changelog for v0.7.2 on GitHub →

FlashInfer v0.7.2 brings MoELayer experts to Rubin through the cuDNN Frost backend, widens Rubin coverage across MLA, GEMM, linear attention and expert-parallel MoE, and adds fused-epilogue Prims-TS GEMMs on B200, B300 and Rubin. On Blackwell datacentre parts, DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and speculative-decode verification picks cuDNN automatically; expert-parallel MoE adds Hopper BF16 and SM120 W4A8 backends.

MoELayer experts run on Rubin

MoELayer on Rubin SM107 can run its experts on cuDNN Frost, a CuTe DSL grouped-GEMM backend for BF16, MXFP8, NVFP4 and MXFP8-activation/MXFP4-weight experts; the BF16 kernels also run on Blackwell SM120. Autotuning admits Frost alongside the existing backends, and CudnnFrostBf16Config, CudnnFrostMxfp8Config, CudnnFrostNvfp4Config and CudnnFrostMxfp8Mxfp4Config in flashinfer.fused_moe select it explicitly. Frost needs PyTorch 2.14 (CUDAGraph.retain_object) and, on SM107, a PTXAS or CuTe DSL assembler from CUDA 13.4 or newer.

  • #6035 Yanqinz/move frost out from experiment
  • #5762 Sm120 cudnn frost moe bf16

DeepSeek-V4 and Kimi K3 serving kernels run on Rubin

BatchMLAPagedAttentionWrapper(backend="auto") plans on B300 and Rubin, and DeepSeek-V4 sparse MLA runs on SM107. Rubin also runs the CuTe DSL GEMM family, mm_mxfp8(backend="cute-dsl") on an SM107-specific kernel, the CuTe packed Kimi Delta Attention decode fast path, the CuTe DSL Mamba2 SSDCombined prefill, hardware stochastic rounding in the Mamba selective state update, and silu_and_mul_mxfp8_quantize. SM107 MegaMoE through MoEEpLayer adds MXFP4 weights with MXFP8 activations, prequantized NVFP4 and MXFP8 checkpoints, and Kimi K3’s SiTU activation, functionally validated on Rubin.

  • #5984 Enable MLA auto-selection on SM103 and SM107
  • #5391 feat(mla): enable TRTLLM-GEN DSv4 sparse MLA/RopeQuant on SM107 (Rubin)
  • #5394 feat(gemm): enable TGV, CuTe DSL low-latency block-scaled and contiguous-grouped GEMMs on SM107 (Rubin)
  • #6136 fix(gemm): use the SM107 CuTe-DSL kernel for mm_mxfp8(backend="cute-dsl") on Rubin
  • #5396 feat(kda,mamba): enable CuTe packed KDA decode, cvt.rs stochastic rounding and Mamba2 SSD on SM107 (Rubin)
  • #5267 perf(mamba): enable native SM107 stochastic rounding
  • #5392 feat(quant): enable fused gated MXFP8 quantization (SwiGLU + MXFP8) on SM107 (Rubin)
  • #5699 feat(moe_ep): support Rubin MXFP4 weights with MXFP8 activations
  • #5662 feat(moe_ep): add Rubin SiTU, prequantized weights, and NVFP4 scaling

GEMMs fuse SwiGLU and QKV-norm-RoPE epilogues on B200, B300 and Rubin

flashinfer.prims_ts adds dense FP8 and NVFP4 GEMMs with fused SwiGLU (fp8_linear_swiglu, fp4_linear_swiglu) and fused QKV + QK-norm + RoPE (fp8_qkv_qknorm_rope, fp4_qkv_qknorm_rope) epilogues on B200, B300 and Rubin SM107. The NVFP4 SwiGLU and QKV variants accept an opt-in tmem_overlap=True tactic, which the autotuner also considers. The module needs the CuTe DSL experimental primitives shipped with nvidia-cutlass-dsl.

  • #5758 feat(prims-ts): add FP8/NVFP4 GEMMs with SwiGLU and RoPE fusion
  • #6255 feat(prims-ts): allow TMEM overlap for NVFP4 SwiGLU epilogues
  • #6286 feat(prims-ts): allow TMEM overlap for NVFP4 qkv overlap

DeepSeek-V4 NVFP4 sparse MLA reaches B200 and B300 and adds prefill on RTX PRO 6000

DeepSeek-V4 serving with an NVFP4 KV cache runs sparse-MLA decode on B200 and B300 through trtllm_batch_decode_sparse_mla_dsv4(kv_cache_format="nvfp4", backend="cake"). On RTX PRO 6000 Blackwell and RTX 5090 the same entry point gains a backend="cake" decode and prefill path for the NVFP4 cache and a decode path for the DeepSeek-V4.1 mixed FP8/FP4 cache (SM120 and GB10/SM121), plus fused RoPE, NVFP4 quantize and paged-insert cache writers. These backends are opt-in.

  • #6103 feat(cake_dsv4): NVFP4 DeepSeek-V4 sparse-MLA decode on SM100/SM103 (backend="cake")
  • #5914 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 sparse-MLA decode backend for RTX PRO 6000 / RTX 5090 (#5690)
  • #5959 feat(cake_dsv4_sparse_mla): add SM120 DeepSeek-V4 NVFP4 sparse-MLA prefill backend for RTX PRO 6000 / RTX 5090
  • #5983 feat(cake_dsv4_sparse_mla): SM120/SM121 DeepSeek-V4.1 mixed-cache sparse-MLA decode (528 B FP8 SWA + 288 B FP4 compressed, BF16 Q)
  • #6033 feat(cake_dsv4): SM120 DeepSeek-V4 NVFP4 fused GPT-J RoPE + quantize + paged-insert writers

Speculative-decode verification runs faster on B200 and B300

BatchDecodeWithPagedKVCacheWrapper(backend="auto") on B200 and B300 routes multi-token decode rows (two to four query tokens per request, as in speculative or multi-token-prediction verification) of fp16/bf16 head-dim-128 grouped-query-attention models to cuDNN, which lowers verification decode latency. The route needs nvidia-cudnn-frontend[cutedsl]>=1.31.0. Single-token decode and runs with explicit determinism controls stay on FA2; FLASHINFER_DECODE_AUTO_CUDNN=0 pins FA2.

  • #5508 feat(cudnn): backend="auto" takes cuDNN for the multi-token paged-decode rows its default FROST engine serves faster (cudnn-frontend 1.30+)

Expert-parallel MoE comes to H100 in BF16 and to SM120 in W4A8

H100 serves an expert-parallel MoE round entirely in BF16 through MoEEpLayer with Sm90_Bf16_Bf16_Bf16_PushCake_MegaMoeConfig: BF16 dispatch, fused SwiGLU, a deterministic combine over a pre-reduced BF16 wire, CUDA Graph capture and up to 32 ranks on one node. SM120 workstations get a W4A8 MegaMoE backend (Sm120_Mxfp4_Mxfp8_Bf16_Cutedsl_MegaMoeConfig) with CUDA Graph replay and uneven token counts across ranks. SplitConfig(comm=..., kernel=...) picks an NVLink one-sided, CAKE or NVLink two-sided communication backend, or the NCCL-EP or NIXL-EP transports, independently of the expert kernel.

  • #5958 feat(cake_mega_moe): SM90 native BF16 MegaMoE expert compute through MoEEpLayer (sm90_bf16_bf16_bf16_push_cake)
  • #6058 feat(cake_mega_moe): pre-reduced BF16 combine wire and fixed-cost levers for the SM90 MegaMoE backend
  • #4632 [SM120] Add W4A8 CuTeDSL MegaMoE backend
  • #5532 refactor(moe_ep): add a MoE-level communication interface for EP dispatch/combine

⚠️ Upgrade notices

  • Install nvidia-cudnn-frontend>=1.31.0. The cuDNN linear-attention backend and the automatic cuDNN decode route need the cutedsl extra: pip install -U 'nvidia-cudnn-frontend[cutedsl]>=1.31.0'.
  • After upgrading flashinfer-python, run flashinfer download-kernels to replace mismatched flashinfer-cubin and flashinfer-jit-cache wheels; --sm sm90a (repeatable) limits it to named architectures.
  • Rename callers of cutlass_fused_moe_prepare_workspace to cake_fused_moe_prepare_workspace, replace prepare_bgmv_moe(..., backend="blackwell") with backend="cake", and import the cuDNN Frost MoE configs from flashinfer.fused_moe (the flashinfer.experimental path is a one-release shim).
  • Sampling: batch-length seed and offset tensors apply one value per output row; callers passing distinct per-row values sample differently from before.

⚠️ Upgrade notices (Experimental)

  • flashinfer.source_mega_moe (MegaMoEPlan, prepare_mega_moe) is removed; use flashinfer.mega_moe_v3.
  • prepare_msa_nvfp4_sparse_decode drops workspace_buffer; msa_nvfp4_decode_workspace_size, workspace_layout and bind_decode_payload are removed. DeepGEMM-family prepare_* entry points drop their descriptor arguments.

v0.7.1

Full changelog for v0.7.1 on GitHub →

FlashInfer v0.7.1 enables quantized expert-parallel MoE on Rubin and Blackwell SM12x, adds fused Kimi K3 speculative verification, and adds cuDNN linear-attention prefill on Blackwell. Sparse MLA gains DeepSeek-V4.1 dual-cache support and compact GLM53-NoPE caches, alongside MLA API improvements and per-model NVFP4 quantization recipes.

Run quantized expert-parallel MoE on Rubin and Blackwell SM12x

Rubin SM107 serves NVFP4 and MXFP8 MegaMoE experts through MoEEpLayer, with BF16 combine and output. On Blackwell SM120/SM121, the W4A16 fused MoE shards experts across GPUs through expert_map, so multi-GPU workstations such as RTX PRO 6000 pairs can split a model’s experts between cards. New CuTe DSL NVFP4 W4A16 and W4A4 MoE backends on SM12x read one prepared expert layout, so a serving stack can choose W4A16 for decode and W4A4 for prefill from a single weight copy; on DGX Spark the W4A16 backend runs a Qwen3.8-Flash-Next MoE layer 1.36× faster than Marlin at one token. The split MoE layer also supports CUDA Graph capture with both NCCL-EP and NIXL-EP.

  • #5384 feat(moe_ep): add SM107 block-scaled MegaMoE backends
  • #4302 feat(moe): expert parallelism for the SM12x W4A16 fused MoE
  • #5735 Canonical-layout NVFP4 W4A16 MoE in CuTe DSL on SM12x
  • #5863 Canonical-layout NVFP4 W4A4 MoE in CuTe DSL on SM12x
  • #5287 feat(moe_ep): CUDA-graph support for the split layer (nccl_ep + nixl_ep)

Kimi K3 adds packed speculative decoding and NVFP4 SiTU experts

Kimi K3 integrations can use packed_fused_kda_decode on Blackwell SM10x to fuse short convolution, Kimi Delta Attention, and gated RMS normalization during speculative verification. In vLLM on eight B300s with six-draft-token speculative decoding, the fused path raised total output throughput by 1.8–3.3% at concurrencies 1 to 32. The API supports ragged batches, accepted-token cache updates, CUDA Graph capture, and FP32 or BF16 recurrent state. NVFP4 MegaMoE also supports Kimi K3’s SiTU activation on Blackwell SM10x.

  • #4809 feat(kda): add fused speculative decode kernel
  • #5503 feat(kda): support bf16 state in packed fused KDA decode for T>1
  • #5455 feat(moe_ep): Support Kimi K3 SiTU in NVFP4 MegaMoE

Kimi Delta Attention and Gated DeltaNet prefill on Blackwell are supported with cuDNN’s open-source kernels

Kimi Delta Attention and Gated DeltaNet prefill run on cuDNN’s open-source fused linear-attention engine on Blackwell SM100, selected with recurrent_kda(backend="cudnn") and chunk_gated_delta_rule(backend="cudnn"). The cuDNN path also runs on CUDA 12 builds, where the default Blackwell Gated DeltaNet prefill kernel requires CUDA 13. New chunk_gated_delta_product and chunk_gated_delta_rule2 entry points add Gated Delta Product and Gated Delta Rule 2 prefill on the same engine.

  • #4968 Add cuDNN backend for linear attention

Run DeepSeek-V4.1 dual-cache attention on SM12x with 19.5% smaller GLM53-NoPE caches

DeepSeek-V4.1 sparse MLA runs on SM120/SM121 with a primary FP8 cache and an optional FP4 or FP8 extra cache, each with its own page size and top-k selection, in FP8 or BF16 compute. GLM53-NoPE caches can use compact 528-byte rows, cutting per-token KV storage by 19.5% against the 656-byte padded layout; existing 656-byte pools keep working through strided cache views.

  • #5197 refactor(sparse-mla): unify SM120 execution, calibration and DSv4.1 support
  • #5075 feat(sm120): runtime KV row stride + canonical 528B GLM53_NOPE rows + NaN-safe masked gathers in sparse MLA

Call MLA prefill directly and reuse execution plans

trtllm_prefill_with_kv_cache_mla, available through both flashinfer.mla and flashinfer.prefill, gives integrations a named prefill entry point with the existing MLA decode API’s signature and behavior. Multi-token prefill callers must explicitly select a compatible backend; XQA supports one query token. BatchMLAPagedAttentionWrapper prepares TRTLLM-GEN, XQA, and monolithic or modular CuTe DSL backends in plan() and reuses them in run(), giving integrations a shared lifecycle for repeated execution and CUDA Graph capture. On SM100, callers can pass backend="auto" and let plan() choose among all available backends for the planned shapes and features.

  • #5158 fix: add discoverable MLA prefill API
  • #5070 feat(mla): add planned TRTLLM-GEN, XQA, and CuTe DSL backends
  • #5463 feat(mla): MLA SM100 auto mode

NVFP4 recipes can be configured per model

NVFP4 quantization APIs and the unified MoE QuantConfig accept an explicit NVFP44Over6Config, so a serving framework can enable 4over6 from a model’s own quantization config and run models with different recipes in one process. The process-wide environment variables remain the default when no recipe is given.

  • #5152 feat(quantization): expose NVFP4 4over6 recipe through the public API

Upgrade notices

  • Install nvidia-cudnn-frontend>=1.30.0. The cuDNN linear-attention backend needs the cutedsl extra: pip install -U 'nvidia-cudnn-frontend[cutedsl]>=1.30.0'.
  • SM90 CuTe DSL MoE: cute_dsl_fused_moe_bf16 takes tactic for tuning, and CuteDslBf16MoEWrapper reorders its constructor arguments. Pass output_dtype, enable_pdl, and use_fused_finalize by keyword. moe_a2a_dispatch takes recv_view_cache by keyword only.

Upgrade notices (Experimental)

  • Update calls to fp8_paged_mqa_logits and fp4_paged_mqa_logits: use block_tables, seq_lens, and max_seq_len; pass the page table before sequence lengths in positional calls; remove num_epi_subtiles. FP4 callers must also rename sf_q to q_sf.
  • Review PrimTS integrations against the experimental API contract. Migrate callers of the removed prims_ts_batch_decode_with_kv_cache and prims_ts_batch_mla_decode_with_kv_cache entry points to the corresponding paged decode wrappers.

v0.7.0

Full changelog for v0.7.0 on GitHub →

FlashInfer 0.7.0 makes the unified mixture-of-experts (MoE) API official, brings TRT-LLM Gen MoE kernels into readable Python source with PrimTS, and introduces Autotuner v2 and a formal experimental-API policy. It also expands sparse and linear attention, expert-parallel serving, and diffusion workloads across Blackwell GPUs.

Read the v0.7 overview and the accompanying deep dives on MegaMoE, Autotuner v2, and the experimental path.

Unified MoE API is official

MoELayer is now an official FlashInfer API, with the lower-level kernel entry points supported alongside it. QuantConfig gives weights, activations, and output explicit QuantFormat fields: QuantConfig(weight=QuantFormat.MXFP4) selects MXFP4 weights with BF16 activations, while adding activation=QuantFormat.MXFP8 selects W4A8. Backend coverage expands with quantization-specific CUTLASS runners, cuTile BF16 and NVFP4 MoE, and CuTe-DSL BF16 MoE on Hopper.

  • #5200 feat(moe): tag the unified MoE API as official with @flashinfer_api
  • #4952 feat(moe): split QuantConfig into weight/activation/output format axes
  • #5061 feat(moe): remove QuantVariant and dispatch QuantConfig by MMA pair
  • #4610 feat(moe): add remaining CUTLASS unified MoE runners
  • #4646 feat(moe): add cuTile fused MoE backend for BF16 and NVFP4 Unified MoE
  • #4878 feat(moe): add CuTe DSL SM90 BF16 MoE backend

TRT-LLM Gen MoE kernels as Python source with PrimTS

The experimental PrimTS MoE backend exposes kernels built with the CUTLASS DSL Primitives and Task Scheduling APIs, making the expert GEMMs available as readable Python source. It supports BF16, per-tensor and block-scaled FP8, and NVFP4/MXFP4 combinations, while reusing TRT-LLM Gen routing and finalization. Accuracy qualification targets B200; B300 qualification is still pending, and supported configurations have backend-specific restrictions. PrimTS attention also gains paged block-sparse attention, variable-window attention, and a unified plan()/run() contract for reusable wrappers.

  • #4361 Add CUTLASS Primitives and Task Scheduling Blackwell MoE kernels
  • #4474 feat(attention): add PrimTS Q64/KV256 and paged GQA block-sparse attention
  • #4599 Adding var-window FMHA context support for PrimsTS.
  • #4829 refactor(attention): unify PrimTS plan/run and page-table contracts

Autotuner v2 tunes the way you serve

autotune_v2() lets applications measure candidates in eager or CUDA-graph execution and persist the results in an environment-specific cache managed by FlashInfer. Atomic cache entries support concurrent rank writes, and autotune_v2_reload() lets homogeneous ranks converge on shared results. In the reported vLLM validation, Qwen3-8B-FP8 at TP2 on B200 reduced its tuning window from 128 seconds on a cold start to 1 second on restart with the same cache; total startup was 410.7 seconds and 165.3 seconds, respectively. Applications opt in through the new API; autotune() remains available.

  • #3861 feat(autotuner): Autotuner v2 (autotune_v2) — managed persistence, deployment-matched measurement, runner contract

Experimental APIs and backends have an explicit opt-in path

Experimental APIs are marked with @flashinfer_experimental_api, and experimental backends have a dedicated flashinfer.experimental namespace. Calling an experimental API or explicitly selecting a marked backend emits a warning; automatic selection includes marked experimental backends only when FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1 is set. The policy defines admission and graduation criteria and keeps experimental implementations JIT-only, outside prebuilt packages.

  • #4880 feat: add experimental API/backend policy, `@flashinfer_experimental_api`, and `flashinfer.experimental` namespace

Expert-parallel MoE expands across Blackwell

moe_ep adds an unquantized BF16 MegaMoE backend and a W4A8 split backend on B200, with MXFP8-packed dispatch for the latter. RTX PRO 6000 and DGX Spark gain an MXFP8 MegaMoE backend with functional correctness validated and performance tuning ongoing. The NCCL-EP split path supports CUDA-graph capture through a reusable handle with per-step update(), validated through vLLM on four B200 GPUs.

  • #4386 feat(moe_ep): SM100 BF16 CuTeDSL MegaMoE kernel
  • #4529 feat(moe_ep): SM100 W4A8 (MXFP8xMXFP4) CuTeDSL split kernel backend with MXFP8 packed dispatch
  • #4387 feat(moe_ep): SM120 MXFP8 swap-AB CuTeDSL MegaMoE kernel
  • #4795 feat(moe_ep): make the nccl_ep split path CUDA-graph capturable

DeepSeek-V4 and MiniMax-M3 sparse attention

Blackwell gains paged FP8/MXFP4 indexer logits and more top-K choices, including a CUB backend with variable-length support. DeepSeek-V4 Flash sparse MLA supports an NVFP4 KV cache on SM120/SM121 through kv_cache_format="nvfp4". MiniMax-M3 sparse attention gains source-distributed CAKE-generated kernels on SM100/SM103; the B200 kernel benchmark reports a 2.45× geometric-mean speedup over the MiniMax baseline across 11 comparable prefill, decode, speculative, and boundary cases.

  • #4365 Add paged MQA logits (attn_scores) kernels for Blackwell SM100
  • #4811 feat(topk): self-sampling GVR V2 backend (SM100/103/107), oracle-tracking auto, and the V1 threshold repair
  • #4621 Enable top_k_varlen on Rubin (SM107) and add the DKG filtered-radix backend (`radix_filter`)
  • #4442 feat: CUB `DeviceBatchedTopK` top-k backend with variable-length support
  • #4955 feat(sm120): add NVFP4 sparse MLA support for DeepSeek V4 Flash
  • #4355 feat(cake_msa): add Blackwell minimax sparse attention source kernels

Linear attention for Kimi K3, Qwen 3.6, and Nemotron-H

Kimi K3 gains a speculative-verification entry point compatible with vLLM’s recurrent verifier, plus CuTe-DSL recurrent prefill on B200/B300 and SM120. The experimental RecurrentKDAPrefillWrapper supports planning and graph-safe prefix checkpoints. Qwen 3.6 gains experimental fused GDN decode steps on SM120 that combine projection, convolution, gating, and recurrent state updates. Nemotron-H gains source-built Mamba SSD-combined and selective-state-update backends on B200/GB300.

  • #4709 feat(kda): output-only KDA decode + drop-in for vLLM Kimi K3 recurren…
  • #4605 feat(kda): add CuTe DSL recurrent prefill backend
  • #4633 feat(kda): add SM120a CuTe DSL prefill backend
  • #5021 feat(kda): support graph-safe CuTe DSL prefix checkpoints
  • #5040 refactor(kda): stop 0.7 advertising unreleased KDA surface (mark prefill wrapper experimental, trim kda_kernels.__all__)
  • #4481 Add Qwen fused GDN decode step for sm120
  • #4708 (perf) add fused_GDN_step support for Qwen 3.6 35B A3B on sm120
  • #4576 feat(cake_mamba): add Blackwell Mamba SSDCombined
  • #4616 feat(cake_mamba): add Blackwell selective state update backend

Diffusion and video attention on Blackwell

Video Sparse Attention gains generated SM100/SM103 kernels, and the block-64 path adds a native CuTe-DSL implementation with Sage FP8 support. SM120 gains a generated Sage block-sparse backend and an optimized NVFP4 attention path for Cosmos workloads. MiniMax-H3 gains a prepared MXFP8 pre-attention pipeline on B200/B300, while PrimTS block-sparse attention adds proxy compensation for Sol-Attn workloads.

  • #4593 feat(cake_vsa): add optimized SM100/SM103 block-sparse attention (VSA)
  • #4612 feat(cute_dsl): replace SM100 blk64 BSA C++/CUTLASS kernel with CuTe-DSL
  • #4951 feat(cake_sage): add SM120 block-sparse attention backend
  • #4502 perf(sm120): optimize NVFP4 attention with N64 score-slot reuse
  • #5060 feat(cake_kernel): add MiniMax-H3 MXFP8 pre-attention for SM100a/SM103a
  • #4872 feat(prims-ts): support proxy-compensated block-sparse attention

PcieIpcAllReduceWorkspace adds an intra-node CUDA-IPC all-reduce for two, four, or eight ranks on PCIe machines without NVLink. Blackwell fused all-gather matmul gains a backend="cake" option and a prepared callable for packed-QKV workloads, with tensor parallelism up to eight GPUs.

  • #4393 [feat]custom all reduce kernel
  • #4722 feat(cake_comm): Add a Cake Blackwell all-gather matmul backend
  • #4822 feat(cake_comm): extend Blackwell all-gather matmul to TP8

Notice: packaging and dependencies

flashinfer-jit-cache is now a small shim that depends on architecture-specific provider wheels. The usual installation command installs the provider set for the selected CUDA and CPU platform; to reduce image size, install only the provider needed for your GPU. If you mirror or vendor wheels, include the provider wheels as well as the shim, and keep their CUDA-specific versions aligned.

Dependency 0.6.18.post1 0.7.0
nvidia-cudnn-frontend >=1.25.0 >=1.29.0
apache-tvm-ffi >=0.1.6,!=0.1.8,!=0.1.8.post0,<0.2 >=0.1.11,<0.2
NCCL-EP Python packages nccl4py>=0.3.1 nccl4py>=0.4.1 and nccl-extensions>=0.1.0
nvidia-cutlass-dsl through [cu12] / [cu13] >=4.6.2a0 >=4.7.0a0

The base nvidia-cutlass-dsl requirement remains >=4.6.2a0; the CUDA extras have the higher floor. Environments pinned to CUTLASS DSL 4.6.2 need compatible dependency pins before installing those extras.

  • #4514 feat(jit-cache): split cache into architecture provider wheels
  • #5096 ci(jit-cache): enable provider-only wheel releases
  • #5182 build: raise the nvidia-cudnn-frontend floor to 1.29.0
  • #5016 Nccl extensions dependency update

Notice: small-batch FP8 groupwise GEMM

⚠️ The CUTLASS gemm_fp8_nt_groupwise path on SM100/SM103 can intermittently produce incorrect output for M <= 32 with scale_granularity_mnk=(1, 128, 128). This pre-existing issue remains in 0.7.0; see #4396 for the investigation. Validate affected workloads before deployment. A different backend requires its own supported shapes and scale layout; there is no documented switch to disable only this CUTLASS fast path.

Notice: API removals and behavior changes

Update callers of the removed APIs before upgrading:

Removed Replacement
comm.trtllm_custom_all_reduce comm.trtllm_allreduce_fusion
comm.trtllm_create_ipc_workspace_for_all_reduce comm.trtllm_create_ipc_workspace_for_all_reduce_fusion
BatchDecodeMlaWithPagedKVCacheWrapper mla.BatchMLAPagedAttentionWrapper
No-op end_forward() methods on decode, prefill, sparse, cascade, and POD wrappers Remove the call
fused_moe.QuantVariant, QuantConfig(variant=...), QuantConfig.from_variant QuantConfig(weight=..., activation=...) using QuantFormat

mamba.checkpointing_ssu replaces the per-step old_x, old_B, old_dt, old_cumAdt, and cache_buf_idx arguments with ring-buffer caches (x_cache, B_cache, dt_cache, ring_start). Update cache allocation and argument binding; the caller-provided out tensor remains required.

For the TRT-LLM backend of BatchPrefillWithRaggedKVCacheWrapper, skip_all_rows_active_check defaults to True. Calls on the default fast path must have positive query and KV lengths for every row; pass False to request device-derived checking when CPU length mirrors are absent.

gated_delta_rule_mtp still resolves an omitted disable_state_update to True in 0.7.0. Pass disable_state_update=True or False explicitly to select the intended behavior and suppress the stale warning announcing a change in this release.

  • #5218 chore: remove deprecated APIs eligible at the 0.7.0 boundary
  • #5061 feat(moe): remove QuantVariant and dispatch QuantConfig by MMA pair
  • #4545 Add ReplaySSM autotuning
  • #5039 fix(attention): default skip_all_rows_active_check to true

v0.6.18

Full changelog for v0.6.18 on GitHub →

This release completes NVIDIA Rubin (SM107) support, brings whole-layer expert-parallel MoE to Hopper, adds decode paths for DeepSeek-V4 sparse attention and Kimi K3 linear attention, and broadens low-precision MoE coverage with MXFP4 on Blackwell RTX PRO and DGX Spark and weight-only W4A16 on B200 and B300.

SM107 (Rubin) support

FlashInfer 0.6.18 completes support for NVIDIA Rubin (SM107), begun in 0.6.16. Rubin devices now dispatch through the same unified APIs as Blackwell — attention, GEMM, MoE, and quantization.

Attention gains trtllm-gen FMHA for SM107, including sparse compression and FP16 softmax. The PrimTS attention path accepts Rubin as well. On the GEMM side, batched and low-latency GEMM both run on Rubin, and CUTLASS NVFP4 SVDQuant is enabled. A CuTe-DSL kernel family specialized for Rubin ships alongside them. trtllm-gen MoE now selects valid tactics on Rubin, and router GEMM and topk_varlen accept SM107.

  • #4526 feat: CuTe DSL kernels for Rubin (SM107) and batched FP8 GEMM for Blackwell
  • #4596 feat: trtllm-gen FMHA features for sm107 (spcompress, fp16softmax)
  • #4509 gemm: enable CUTLASS NVFP4 SVDQuant on SM107
  • #4710 fix: support SM107 (Rubin) in topk_varlen, FlashKDA, and router GEMM capability gates
  • #4792 [release-v0.6.18] Five SM107 (Rubin) fixes: 133 CI failures

Expert-parallel MoE runs whole-layer on Hopper

The moe_ep mega-kernel stack was Blackwell-only. Two Hopper FP8 backends, Sm90PullFp8MegaMoeConfig and Sm90PushFp8MegaMoeConfig, now let Hopper deployments run dispatch, FC1, SwiGLU, FC2, and combine as one fused layer behind the existing MoEEpLayer, instead of composing an NCCL all-to-all with a local fused-MoE operator. The pull backend reaches 562 TFLOPS/rank at a 384-expert DeepSeek-class geometry. The push backend supports CUDA Graph capture, and its opt-in fused FC1 epilogue drops an approximately 1 GiB per-rank activation buffer at the DeepSeek-V3 EP8 shape. Its grouped GEMM requires CUDA Toolkit 12.8 or newer.

  • #4113 feat(moe_ep): SM90 (Hopper) pull-style FP8 mega-MoE backend
  • #4069 feat(moe_ep): add SM90 push FP8 mega-MoE backend for Hopper
  • #4449 refactor(moe_ep): taxonomy/provenance restructure; incorporate SM90 push-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix

HCA decode backend and top-K selection for DeepSeek-V4

Sparse attention picks the top-K KV positions per request on every decode step, then attends over the compressed cache; both halves now have dedicated paths. FP8 Heavily Compressed Attention (HCA) arrives for SM100/SM103 via trtllm_batch_decode_sparse_mla_dsv4(..., backend="cute-dsl"), taking arbitrary sliding-window row order including ring rotation and wraparound while keeping the compressed cache paged. The new flashinfer.top_k_varlen handles ragged batches through a Blackwell radix kernel, a guess-verify-refine kernel that warm-starts from the previous step’s indices, and a CUTLASS fallback for any GPU. SM120/121 also picks up top-k 192 and 256.

  • #3943 Add CuTe DSL HCA backend for DeepSeek V4 sparse MLA
  • #3901 feat(topk): Add top_k_varlen with GVR and radix backends for sparse-attention KV selection
  • #4380 feat(sm120): consolidate DSV4 sparse MLA top-k 192/256 support

Kimi K3 decode fuses into one Blackwell kernel

flashinfer.fused_kda_decode folds Kimi K3’s width-four depthwise causal convolution, SiLU, recurrent Kimi Delta Attention update, and gated RMSNorm into a single SM100 launch, covering the production head_dim 128 shapes at 12, 24, 48, and 96 heads and updating the convolution cache and FP32 state in place. On B200 under CUDA Graphs it is 1.33x the vLLM fused kernel at one row (table geomean 1.13x). A T=1 fast path inside recurrent_kda and shared SM100-family recurrent kernels round out the KDA stack.

  • #4243 feat(kda): add fused Kimi K3 decode kernel
  • #4417 feat(kda): add packed-input CuTe decode kernel
  • #4562 feat(cake_kda): add optimized H12 packed decode across SM100 family

MXFP4 MoE and video sparse attention on Blackwell RTX PRO

MXFP4 checkpoints run natively on SM120/121: b12x_fused_moe and B12xMoEWrapper accept quant_mode="mxfp4" across the existing fused schedules including CUDA Graph reuse, the b12x dense GEMM gains the matching path, and on an RTX PRO 6000 Blackwell Server Edition MXFP4 tracks NVFP4 against a strict quantized reference. Video Sparse Attention, until now datacenter-Blackwell only, reaches these parts through a vsa_sm120_blk64 backend on BlockSparseAttentionWrapper. Gemma 4 gains asymmetric VO-split NVFP4 paged prefill on SM120/121.

  • #4290 feat(sm120): add native MXFP4 W4A4 fused MoE
  • #4479 feat(b12x): support MXFP4 dense GEMM on SM120
  • #4259 integrate SM120 VSA (Video Sparse Attention) block-sparse backend
  • #3684 feat(attention): asymmetric VO-split NVFP4 paged prefill (qk=512/vo=256) for Gemma-4 on SM120/121

W4A16 MoE and dense GEMM extend to B200 and B300

Weight-only NVFP4 against BF16 activations, which 0.6.14 shipped for SM12x, now runs on the SM100 family: CuteDslMoEWrapper and cute_dsl_fused_moe_nvfp4 accept quant_mode="w4a16", decoding weights to BF16 inside the kernel so no separate activation-quantization or repack launch is needed, and mm_bf16_fp4 gains a dedicated SM100/SM103 kernel. Consuming BF16 directly pays off where MoE decode is memory-bound: at a DeepSeek EP8 shape on B200, W4A16 is 1.50x the W4A4 baseline at one token; W4A4 still wins large-batch prefill. GeGLU-tanh and SiTU are supported.

  • #4048 feat(moe): support BF16 activations in CuTe DSL NVFP4 MoE
  • #4466 feat: sm100 cute_dsl w4a16 gemm
  • #4394 feat(moe): support GeGLU and SiTU in W4A16

Unified MoE API adds shared experts, MXINT4, and CUTLASS runners

Shared experts now work through the unified API rather than low-level kernel entry points, via ExpertConfig.num_fused_shared_experts on the block-FP8 and FP4 runners, completing in the unified API what 0.6.15 and 0.6.17 added for FP8 and FP4. The API also gains MXINT4, CUTLASS BF16 and W4A16 runners on SM90, packed per-tensor FP8 routing, BF16 FromLogits routing, and TopKSigmoid.

  • #4326 Fused shared experts support via the unified API
  • #4320 feat(moe): enable MxInt4 in unified API
  • #4328 feat(moe): add unified CUTLASS BF16 and W4A16 runners
  • #4227 feat(moe): add packed per-tensor FP8 and BF16 FromLogits routing to unified MoE API
  • #4404 feat(moe): add TopKSigmoid routing method (TopK -> Sigmoid)

Fused MNNVL all-reduce tail for tensor-parallel MoE

allreduce_fusion gains a BF16 Blackwell CuTe-DSL backend for the latency-critical tail of tensor-parallel MoE layers, fusing all-reduce, residual add, and RMSNorm, optionally preceded by MoE finalize and shared-expert add, over MNNVL/NVLink multicast. One backend spans decode to prefill by switching protocol with token count, with initial profiles targeting GB300 on TP8 and TP16.

  • #4358 feat(comm): add Blackwell MNNVL CuTe DSL all-reduce fusion backend

Smaller JIT-cache wheels; SM75 and single-request FA2 are JIT-only

The flashinfer-jit-cache wheels no longer ship precompiled kernels for SM75 (Turing). Those GPUs still run; the kernels compile on first use. CUDA 13 AArch64 wheels also drop native SM121a cubins (DGX Spark keeps running via SM120 family cubins), and the single-request single_decode_with_kv_cache / single_prefill_with_kv_cache FA2 modules are no longer AOT-prebuilt — those APIs still JIT. Fatbins use size-oriented compression.

  0.6.17 0.6.18
cu129 x86_64 1.94 GB 1.02 GB
cu130 x86_64 1.51 GB 1.02 GB
cu130 aarch64 1.69 GB 1.13 GB
  • #4757 build(jit): reduce JIT-cache wheel size
  • #4760 chore(aot): exclude single prefill/decode modules from jit-cache prebuilds

v0.6.17

Full changelog for v0.6.17 on GitHub →

This release brings MoE expert parallelism into serving engines, refreshes the Blackwell SM12x fused-MoE kernels with an FP4 accuracy fix, extends the unified MoE API to MXFP4 W4A8/W4A16 and shared experts, and adds decode coverage for Kimi K3 MLA and MiniMax-M3 sparse attention on vLLM.

MoE expert parallelism production-ready in vLLM

The MegaMoE path in flashinfer.moe_ep is ready for serving engines: full CUDA-graph capture and replay, a fused single-launch quantize-and-stage hot path, prequantized weight packs, symmetric-buffer workspaces pooled across layers, and a persistent knob cache that resolves tuned knobs by lookup, so production sessions run with no in-engine autotuning. An opt-in fault-tolerance rank mask, available over both NCCL-EP and NIXL-EP, masks and skips a peer that times out during dispatch or combine, keeping the job alive when a rank dies. Deployment gets simpler too: the CuTe-DSL runtime floor returns to 4.5.2, the version vLLM 0.25.1 pins, and a new BootstrapConfig.device lets the host framework pin each worker’s CUDA device.

  • #4079 feat(moe_ep): MegaMoE framework integration ready: CUDA graph support, fused quant+stage launch, persistent knob cache, and prequantized weight packs
  • #4183 feat(moe_ep): fault-tolerance rank mask (NCCL-EP + NIXL-EP)
  • #4101 perf(moe_ep): CuTe-DSL 4.5.2 mainloop WAR — drop the 4.6.1 runtime floor
  • #4348 fix(moe_ep): let the host framework pin the CUDA device via Bootstrap

Blackwell SM12x fused MoE refreshed, with an FP4 accuracy fix

W4A4 serving on DGX Spark and RTX PRO parts (GB10, SM120/SM121) now delivers the output quality its benchmark scores imply, with two NVFP4 quantization bugs fixed and a new input_global_scale that lets integrators pass a checkpoint’s weight scale directly. The SM12x fused-MoE families are synced to current b12x: the NVFP4 W4A4 backend reaches kernel parity across decode and prefill, and the W4A16 family adds cooperative persistent launches, a tensor-core decode path for small batches, and shape-stable route packing that keeps decode batch-size changes recompile-free.

  • #3932 fix(b12x): correct fp4 quantization numerics and add input_global_scale to decouple weight and activation scales
  • #4285 perf(moe): sync SM12x NVFP4 fused-MoE kernels to b12x HEAD
  • #4255 feat(moe): sync SM12x W4A16 fused MoE family to b12x HEAD
  • #4253 feat(gemm): sync mm_fp4 SM120 NVFP4 dense GEMM kernel to b12x HEAD
  • #4130 feat(sm120): fused MoE (SwiGLU) via moe_gemm is_gated for cute SM120 groupwise GEMM

Unified MoE API adds MXFP4 W4A8 and W4A16, shared experts, and SiTU

TRTLLM-gen MXFP4 weights now run through the unified MoELayer API against both MXFP8 activations (W4A8) and BF16 activations (W4A16), completing the FP8 series begun in 0.6.16, alongside per-tensor routed FP8. Routing adds an unpacked pre-routed FP4 mode that accepts topk_ids and topk_weights as separate contiguous tensors. TRTLLM-gen FP4 MoE also gains shared-expert fusion, extending to FP4 what 0.6.15 added for FP8, plus SiTU activation for MXFP4 x MXFP8 and NVFP4 x NVFP4.

  • #4159 feat(moe): add TRTLLM MXFP4 W4A8 and W4A16 unified API support
  • #4088 feat: close feature gap by wiring up per-tensor routed FP8 fused-moe
  • #4104 feat(moe): add unified unpacked pre-routed FP4 mode
  • #4239 feat(moe): support shared expert fusion for trtllm-gen fp4 moe
  • #4180 [feat] Add SITU trtllmgen MOE

Kimi K3 MLA decode, and MiniMax-M3 sparse attention under vLLM

Blackwell decode now covers Kimi K3’s MLA geometry — 96 global query heads against one KV head, TP-local head counts down to 6, speculative query lengths up to 8, and context parallelism for long contexts — by packing query-token and query-head rows into shared CuTe-DSL tiles, adding compact variable-length Q, and extending TRTLLM-gen dense and sparse MLA to non-power-of-two head counts. Sparse MLA decode also serves the no-rotary-tail shape (kv_lora_rank=512, qk_rope_head_dim=0) natively. MiniMax Sparse Attention accepts vLLM’s packed paged KV layout for MiniMax-M3, unblocking the vLLM integration on SM120/SM121, with lower per-call overhead on paged decode.

  • #4178 feat(mla): support packed low-head and variable-Q decode
  • #4108 feat: support native qk_rope_head_dim=0 sparse MLA decode in trtllm-gen
  • #4039 feat(msa): accept K/V views split from a packed paged KV cache
  • #4324 MSA decode path improvements

Ulysses sequence parallelism for long-context and video diffusion

Ulysses sequence parallelism gets its head-scatter / sequence-gather all-to-all as a public API, for video diffusion transformers and other long-sequence attention workloads. A fused NVLink-P2P kernel folds the layout permutation directly into the cross-GPU writes over CUDA IPC for a single coalesced push, with automatic NCCL fallback when P2P is unavailable.

  • #3820 Feat/ulysses p2p a2a
  • #4240 docs: improve Ulysses communicator and MoE EP docs

v0.6.16

Full changelog for v0.6.16 on GitHub →

This release delivers the MegaMoE kernels promised for expert parallelism in 0.6.15 and extends Blackwell model coverage — MiniMax-M3 sparse attention on Blackwell RTX and DGX Spark, and a unified MoE API that now executes per-tensor FP8, block-scale FP8, and B12x NVFP4/W4A16 — plus distributed serving under Confidential Computing with multicast-free all-reduce fusion. Under the hood, an on-disk JIT cache for CuTe-DSL kernels cuts cold-start compilation and shrinks the install.

⚠️ FlashInfer 0.6.16.post2 has picked up the latest tvm-ffi compatibility fix.

⚠️ FlashInfer 0.6.16.post4 adds a Python 3.10 compatibility hotfix.

We recommend upgrading to the latest 0.6.16 post-release.

MegaMoE kernels land in expert parallelism (moe_ep)

The MoEEpLayer now unifies the split and mega execution paths under one entry point and adds three mega backends — deep_gemm_mega (FP8/FP4), nvfp4_cutedsl, and mxfp8_cutedsl — each fusing expert-parallel communication with the local MoE in a single symmetric-memory kernel (Blackwell SM100+, NVSHMEM). In a microbenchmark, on a GB200 node (EP=4, DeepSeek-V3-like geometry) the tuned CuTeDSL NVFP4 backend with in-register FP4 combine reaches up to 1.89× the throughput of deep_gemm_mega at 8192 tokens/rank. NIXL-EP transport gaps and combine deadlocks blocking the vLLM Fleet/Handle adapters are also closed.

  • #3852 feat: MegaMoE Kernel integration in moe_ep
  • #3980 feat(moe_ep): restructure MegaMoE kernel sources into kernel_src + latest CuTeDSL MegaMoE kernels with improved performance
  • #4075 feat(moe_ep): close the nixl_ep transport gaps blocking the vLLM Fleet/Handle adapters
  • #4139 fix(moe_ep): fix two NIXL-EP combine deadlocks under real serving load

MiniMax Sparse Attention (MSA) on Blackwell RTX and DGX Spark

MiniMax-M3’s MSA — a proxy/top-k indexer plus sparse prefill, decode, and combine — now runs on Blackwell RTX and DGX Spark (SM120/121) GPUs. The tensor-core kernels are rebuilt on SM12x warp-level mma.sync, top-k and combine are rewritten in CuTe-DSL, and an optional NVFP4 indexer is available, with prefill/decode accepting FP8 or NVFP4 KV (paged or flat). New APIs live under flashinfer.msa_ops.

  • #3655 feat(attn): Enable MiniMax Sparse Attention (MSA) for Consumer Blackwell GPUs (SM120/121)

XQA decode adds sliding-window, attention sinks, and ragged Q for speculative decode

The XQA decode kernel — used on Blackwell RTX and DGX Spark (SM120/121) for models with attention sinks — expands coverage for sliding-window attention and ragged Q, across causal and non-causal draft-block mask modes and combinations of them. Ragged Q lets each request in a batch verify a different number of draft tokens, and sliding-window masking is now computed per draft-token row, making the kernel useful for speculative-decoding workloads across models on SM120/121.

  • #4137 feat(xqa): ragged Q and per-row sliding-window masking for speculative decode

Unified MoE API reaches parity with the legacy FP8 and NVFP4 paths

The unified MoELayer API continues to expand quantization formats to reach parity with legacy flat APIs: TRTLLM per-tensor FP8 (SM100/SM103, with Llama4 routing-scale-on-input), DeepSeek FP8 and MXFP8 block-scale (SM100/SM103), and SM120/SM121 B12x NVFP4 and W4A16 backends. In-kernel routing (FromLogits) is now wired through the unified API and fuzzer, so precomputed and in-kernel routing share one path with CUDA-graph and autotuning coverage.

  • #4091 feat(moe): add unified per-tensor FP8 support
  • #4026 feat(moe): add unified block-scale FP8 support
  • #3983 feat(moe): add B12x backends to unified MoE API and add corresponding tests
  • #3892 feat(moe): wire in-kernel routing (FromLogits) into unified MoE API + routing axes in fuzzer

Confidential Computing: multicast-free all-reduce fusion and FP8 AllReduce

The TRT-LLM AllReduce-fusion workspace now allocates a multicast-free IPC workspace when NVIDIA Confidential Computing is detected (is_confidential_compute(), overridable via FLASHINFER_CONFIDENTIAL_COMPUTE), so one-shot Lamport and two-shot sync fusion run under CC where cuMulticast setup otherwise fails. Separately, a new flashinfer.comm.quantized_all_reduce() halves AllReduce transfer volume by quantizing activations to FP8 before P2P transfer over symmetric memory (SM90+, NVSwitch).

  • #3993 comm: multicast-free (IPC) workspace for TRT-LLM AR fusion under Confidential Computing
  • #3759 feat(comm): Add FP8 quantized two-shot AllReduce via symmetric memory

Faster JIT cold-start and smaller install

CuTe-DSL kernels now persist to an on-disk cache (JitSpec gains a JitSpecCuteDsl backend) and reload via JITLink in about 3–30 ms instead of recompiling in every new process; mm_fp4 autotuning additionally compiles tactics in parallel and reuses the shared disk cache, cutting autotune wall time. Pruning architecture gencode that dispatch can never load removes roughly 1.6 GB of installed size from the CUDA-13 aarch64 JIT-cache wheel.

  • #3874 feat(jit): JitSpec ABC + disk cache for JIT-compiled CuTe-DSL kernels
  • #4029 perf(gemm): Improve mm_fp4 cute-dsl autotune time via disk-cache and parallel compilation
  • #3947 perf(jit): drop dead SM12x gencode from SM10x-serving modules
  • #4073 perf(jit): prune arch gencode that dispatch can never load

v0.6.15

Full changelog for v0.6.15 on GitHub →

This release ships Expert Parallelism (moe_ep) in the default install and extends the TRTLLM-GEN MoE stack for large-model serving. It broadens Blackwell model coverage — Gemma 4 / MiniMax-M3 MoE on consumer / DGX-Spark SM12x, DeepSeek-class MLA decode on B300, and context-parallel GDN on SM120 — and adds Video Sparse Attention plus CUDA-graph-safe FP8 all-reduce fusion for distributed inference.

Unified MoE API with Expert Parallelism (Experimental API to try!)

FlashInfer’s unified MoE compute API is now wired into expert parallelism. A new flashinfer.moe_ep.MoEEpLayer runs one MoE layer split across ranks as dispatch → per-expert grouped GEMM → combine, over pluggable transport (NCCL-EP via nccl.ep/nccl4py, and NIXL-EP). The expert GEMM reuses the unified flashinfer.fused_moe.MoELayer as a pure per-expert grouped GEMM — routing lives in dispatch/combine. moe_ep is part of the default install (CUDA 13+), with vLLM-facing APIs and checkpoint-safe MoE all-to-all graph VAs. This EP API is also the integration surface for MegaMoE kernels in upcoming releases — try it out and let us know how it works for you.

  • #3686 MoE-EP: wire unified MoE compute into NCCL-EP / NIXL-EP expert parallel (LL + HT)
  • #3821 feat: make moe_ep (EP) part of the default install; drop nccl submodule
  • #3813 Feat/vllm moe ep api
  • #3727 feat(mnnvl): preserve MoE all-to-all graph VAs across checkpoint restore
  • #3591 test(moe): align EP-offset tests and docs with global expert ids (gh #3547)

Gemma 4 and MiniMax-M3 NVFP4 MoE now run on Blackwell SM12x

Gemma 4 and MiniMax-M3 NVFP4 MoE now run on Blackwell SM12x (consumer / DGX Spark), enabled by two new NVFP4 MoE activation functions — gelu_tanh and swiglu_oai — added to the SM12x MoE path.

  • #3744 feat(moe): add gelu_tanh and swiglu_oai activations to b12x NVFP4 MoE for SM12x
  • #3737 Feat/cutedsl moe swiglu oai and use activation type to pass in

TRTLLM-GEN MoE adds shared experts, DeepSeek-V4 routing, and low-latency FP8

TRTLLM-GEN MoE gains shared-expert fusion for FP8 paths, hash-based DeepSeek-V4 routing (hash_topk), and a fused FP8 blockwise megakernel that cuts latency for small batches (BS ≤ 8).

  • #2625 feat: Fuse shared experts into trtllm_gen moe (fp8)
  • #3561 feat(moe): add DSv4 hash-based MoE routing (MOE-01-HASH)
  • #3424 Fused FP8 blockwise MoE megakernel For BS <= 8

Video Sparse Attention and DeepSeek MLA decode on Blackwell B300

Video Sparse Attention (VSA) is now integrated into the block-sparse attention API, bringing efficient long-context attention for video diffusion models to FlashInfer. DeepSeek-class MLA decode extends onto Blackwell B300 (SM103) with cluster-aware CUTLASS split_kv, and CuTe-DSL GQA decode adds sliding-window and attention-sink masking for newer attention variants.

  • #3250 integrate VSA (Video Sparse Attention) into block-sparse API
  • #3888 fix(mla): enable cute-dsl/auto on SM103 + cluster-aware cutlass split_kv
  • #3794 feat(cute-dsl): sliding window and attention sinks for GQA decode (non-paged + paged)

Linear attention: context-parallel GDN on SM120 and faster KDA decode

Gated delta-rule (GDN) linear attention adds SM120 context-parallel delta rule support, extending the CuTe-DSL GDN rewrite from 0.6.14 onto Blackwell SM120 prefill, alongside SM90 context-parallel prefill optimizations. KDA recurrent-decode kernels are also optimized for lower decode latency.

  • #3659 feat: add sm120 cp delta rule
  • #3788 perf: optimize sm90 cp delta rule
  • #3766 perf(kda): optimize recurrent decode kernels

Distributed comm: FP8 fusion and CUDA-graph checkpoint restore

Dynamic per-token FP8 quantization fuses into allreduce + residual + RMSNorm (TRT-LLM and MNNVL backends, CUDA-graph safe). All-reduce workspaces support checkpoint_prepare / checkpoint_restore so physical backing can be released and remapped at stable VAs across serving checkpoints.

  • #3421 feat: add dynamic FP8 allreduce RMSNorm fusion
  • #3745 feat(comm): preserve all-reduce graph VAs across checkpoint restore

v0.6.14

Full changelog for v0.6.14 on GitHub →

This release pushes FlashInfer’s coverage onto Blackwell RTX PRO and DGX Spark silicon, lands the CuTe-DSL rewrite of the gated-delta-rule (GDN) kernels behind Qwen3.5/3.6, and makes those kernels production-ready inside vLLM.

⚠️ Manual intervention needed: install instructions changed

Please modify your install instructions as you bump the version to 0.6.14.

pip install flashinfer-python

pip install flashinfer-cubin --index-url https://flashinfer.ai/whl     # this is the difference

pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu129
# OR
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu130

See also #3808.

Gemma 4 and W4A16 on Blackwell RTX Pro and DGX Spark

Blackwell SM12x parts (RTX PRO 6000, DGX Spark / GB10) previously trailed Blackwell GB200 NVL72 (and B300 SM103) on attention shape and quantization coverage. This release closes the gap for Gemma 4 and weight-only-quantized inference: head_dim=512 attention for Gemma 4’s global layers now runs on SM120/121, the FMHAv2 prefill path gains head_dim=256/512 and sliding-window masking on SM120, the new mm_bf16_fp4 W4A16 GEMM is tuned for DGX Spark (completing W4A16 across both dense GEMM and MoE), and gated tanh-GELU brings Gemma 4 MoE onto the CUTLASS backend.

  • #3576 feat(attention): head_dim=512 support for attention prefill & decode for Gemma 4 on SM120/121
  • #3518 FMHAv2 on SM120 for head_dim 256/512 + sliding-window masks
  • #3597 feat: Add BF16_FP4 GEMM with cuDNN and CuTe-DSL backends for SM120/121 for W4A16 workloads
  • #3501 [feat] Add gated tanh-GELU (GeluTanh) activation to CUTLASS fused MoE (GEMMA 4)

GDN / gated delta rule: CuTe-DSL overhaul (Qwen3.5/3.6)

The gated-delta-rule kernels behind the Qwen3.5/3.6 family were rewritten from CUTLASS C++ to CuTe-DSL (#3491). The rewrite eliminates the C++ JIT compilation pain reported by customers and establishes the base for context-parallel delta-rule kernels — covering SM90 prefill and its context-parallel variant, SM120 prefill, and a ~20–25% GDN prefill speedup from mainloop efficiency work.

  • #3477 feat: add sm90 delta rule dsl prefill
  • #3481 feat: add sm90 cp delta rule dsl
  • #3479 feat: add sm120 delta rule dsl prefill
  • #3742 Improve GDN prefill perf by ~20-25% (mainloop efficiency)

GDN production-ready in vLLM

Two gaps blocked GDN serving in vLLM (#3602); both are now resolved. GDN kernels are compilation batch-size agnostic, so a single compiled cubin is reused across batch shapes instead of recompiling on every new batch size at inference time, and a new BF16 state recovery/decode kernel writes SSM state into preallocated space to supply the MTP-compatible spec-decode path vLLM needs.

  • #3649 perf(gdn): make GDN kernels compilation batch-size agnostic (support dynamic batch shapes for vLLM integration)
  • #3502 feat(gdn): BF16 state recovery/decode kernel with per-request K and f…

DeepSeek-class sparse MLA on Blackwell, FP8 KV on Hopper

New sparse-MLA paged-attention kernels extend the DeepSeek-V4 (d_qk=512) and DeepSeek-V3.2 / GLM-5.1 (d_qk=576) families onto SM120/121 through the existing flashinfer.mla APIs, with DSv4 coverage broadened to 8/16/32 head counts. On Hopper SM90, native FP8 KV cache support eliminates SGLang’s per-layer cast workaround, saving an HBM round-trip per layer on DeepSeek-V3/V4 while staying bit-identical to the BF16 path.

  • #3395 feat(attention): add SM120 sparse MLA kernels
  • #3545 Support smaller DSv4 sparse MLA head counts
  • #3694 feat(attention): FP8 KV cache support for Hopper SM90 MLA