The problem

A Mixture of Experts (MoE) architecture replaces standard dense feed-forward networks with a set of sparse sub-networks, and dynamically routes individual tokens to a top-K subset of experts. This allows the model to have large scaling in terms of weight at sub-linear computational scaling. But, large amounts of weights often do not fit into a single GPU. Serving large Mixture-of-Experts (MoE) models, hence relies on expert parallelism: each forward pass dispatches tokens to their assigned experts across GPUs, executes grouped GEMMs, and combines the outputs. Traditional implementations perform dispatch, quantization, GEMMs, and combine as separate stages, incurring repeated kernel launches, synchronization, and memory traffic. Communication and computation also tend to execute sequentially, leaving hardware resources underutilized.

Our approach

FlashInfer’s MegaMoE implementation fuses an MoE layer into a single persistent kernel per forward pass. The kernel performs dispatch, activation quantization, both grouped GEMMs, and result combination while overlapping communication and computation through symmetric (NVSHMEM) memory.

The implementation lives in flashinfer.moe_ep behind a single layer API with interchangeable backends, for eg.:

  • NVFP4 (W4A4) — CuTe DSL implementation with 4-bit weights and activations using block-16 scaling
  • MXFP8 (W8A8) — CuTe DSL implementation with 8-bit weights and activations using block-32 scaling
  • DeepGEMM (W4A8) — integration of the existing fused DeepGEMM kernel, serving as both the baseline and fallback implementation

The combine-phase communication format can be configured as BF16, MXFP8, or NVFP4, allowing bandwidth to be traded against numerical precision when beneficial for larger workloads.

Kernel tuning is automatic by default through built-in heuristics and a persistent offline-tunable knob cache, with optional first-run autotuning (knobs="auto") or explicit user-provided tuning parameters.

Correctness is validated through three complementary mechanisms:

  • PyTorch reference implementations for all compute paths
  • Bit-exact expert-parallel vs. single-GPU multi-rank tests
  • CUDA Graph capture and replay tests

Usage

These schematic examples show backend selection. Supply the layer parameters, process-group setup, inputs, and backend-specific kernel configuration for your deployment.

from flashinfer.moe_ep import MoEEpLayer, MegaConfig

layer = MoEEpLayer(
    ...,
    backend=MegaConfig(
        megakernel=megakernel_config,
        quantize_input=True,
        preprocess_weights=True,
    ),
)
output = layer.forward(inputs)

Weights are accepted in their standard checkpoint layout. The layer performs quantization, weight reordering, and symmetric-buffer allocation automatically.

In contrast to the MegaConfig, a user can also choose a split config, where the mega kernel configuration cannot fit. For example, because of topology or data type.

from flashinfer.moe_ep import MoEEpLayer, SplitConfig, NcclEpConfig

layer = MoEEpLayer(
    ...,
    backend=SplitConfig(
        comm=NcclEpConfig(),
        kernel=split_kernel_config,
    ),
)
output = layer.forward(inputs)

Performance

Unless otherwise noted, the results below were measured with vLLM 0.25.1 and FlashInfer moe_ep. We report three backends. native is vLLM’s own DeepGEMM MegaMoE implementation and serves as the baseline. fi_dg routes that same DeepGEMM MegaMoE kernel through FlashInfer moe_ep, so its ratio against native isolates integration overhead rather than kernel work. fi_cutedsl is FlashInfer’s NVFP4 CuteDSL MegaMoE backend, which is the new kernel under study.

Native and fi_dg use the original MX checkpoint, while fi_cutedsl uses the NVFP4 cast of the same base weights, so throughput comparisons should be read together with the accuracy gate below.

End-to-end results in vLLM

1x8 SM100 — DeepSeek-V4-Flash, EP8

Throughput comparison for 1x8 SM100 — DeepSeek-V4-Flash, EP8. Native, fi_dg, and fi_cutedsl results for four workloads are reproduced in the table below.
Reported vLLM throughput; higher is better. Gray and charcoal show the MX-checkpoint paths; NVIDIA green shows the NVFP4-checkpoint path. Values are from the table below, not new measurements. Read alongside the accuracy gate.
cell native tok/s fi_dg fi_cutedsl
prefill-8k 38986 40225 (1.032x) 46584 (1.195x)
decode-1k 30845 31494 (1.021x) 32741 (1.061x)
100K ISL / 1K 29632 30230 (1.020x) 32913 (1.111x)
32K ISL / 32 35700 36597 (1.025x) 42028 (1.177x)

Latency on the interactivity-sensitive cells also improves: TTFT drops from 49.3 s to 42.2 s at 100K ISL / 1K and from 15.1 s to 12.8 s at 32K ISL / 32, while ITL p50 improves from 56.1 ms to 51.9 ms and from 226.5 ms to 191.2 ms, respectively.

1x8 SM100 — DeepSeek-V4-Pro, EP8

Throughput comparison for 1x8 SM100 — DeepSeek-V4-Pro, EP8. Native, fi_dg, and fi_cutedsl results for four workloads are reproduced in the table below.
Reported vLLM throughput; higher is better. Gray and charcoal show the MX-checkpoint paths; NVIDIA green shows the NVFP4-checkpoint path. Values are from the table below, not new measurements. Read alongside the accuracy gate.
cell native tok/s fi_dg fi_cutedsl
prefill-8k 15240 15630 (1.026x) 20074 (1.317x)
decode-1k 12897 13157 (1.020x) 15368 (1.192x)
100K ISL / 1K 12223 12453 (1.019x) 15053 (1.231x)
32K ISL / 32 14117 14435 (1.023x) 18250 (1.293x)

The latency gains are larger on the bigger model. At 100K ISL / 1K, TTFT improves from 122.3 s to 95.0 s; at 32K ISL / 32, it improves from 38.4 s to 29.5 s. ITL p50 improves from 134.3 ms to 111.8 ms and from 573.9 ms to 441.3 ms on the two interactivity cells.

Across both models, fi_dg stays near 1.02x, indicating that the moe_ep wrapper adds effectively no measurable overhead in this configuration. The gain for the NVFP4 configuration is represented by fi_cutedsl, which improves throughput by 1.06x to 1.20x on DeepSeek-V4-Flash and by 1.19x to 1.32x on DeepSeek-V4-Pro.

1x4 SM103 — DeepSeek-V4-Flash, EP4/TP4

Throughput comparison for 1x4 SM103 — DeepSeek-V4-Flash, EP4/TP4. Native, fi_dg, and fi_cutedsl results for four workloads are reproduced in the table below.
Reported vLLM throughput; higher is better. Gray and charcoal show the MX-checkpoint paths; NVIDIA green shows the NVFP4-checkpoint path. Values are from the table below, not new measurements. Read alongside the accuracy gate.
cell native tok/s fi_dg fi_cutedsl
prefill-8k 47412 49145 (1.037x) 56432 (1.190x)
decode-1k 32313 32936 (1.019x) 34396 (1.064x)
100K ISL / 1K 35575 36389 (1.023x) 38632 (1.086x)
32K ISL / 32 43615 44846 (1.028x) 50360 (1.155x)

Latency on the interactivity cells also improves: TTFT drops from 40.4 s to 35.2 s at 100K ISL / 1K and from 12.4 s to 10.6 s at 32K ISL / 32, while ITL p50 improves from 47.1 ms to 44.7 ms and from 184.9 ms to 159.0 ms.

The 1x4 SM103 pattern matches the 1x8 SM100 sweep cell for cell: fi_dg stays near wrapper parity, while fi_cutedsl posts the largest gains on the prefill-heavy cells and the smallest gain on decode-1k.

Kernel microbenchmarks

1x8 SM100 — DeepSeek-V4-Flash geometry

tok/rank dg nvfp4 bf16 +ikr +combine_nvfp4 +combine_mxfp8
512 154.7 189.4 (0.82x) 192.0 (0.81x) 168.9 (0.92x) 173.1 (0.89x)
1024 233.5 232.4 (1.00x) 237.0 (0.99x) 197.5 (1.18x) 205.9 (1.13x)
2048 379.1 334.8 (1.13x) 334.8 (1.13x) 273.4 (1.39x) 287.7 (1.32x)
4096 680.0 578.5 (1.18x) 574.4 (1.18x) 422.9 (1.61x) 472.0 (1.44x)
8192 1320.4 1104.8 (1.20x) 1091.7 (1.21x) 772.2 (1.71x) 887.3 (1.49x)

1x8 SM100 — DeepSeek-V4-Pro geometry

tok/rank dg nvfp4 bf16 +ikr +combine_nvfp4 +combine_mxfp8
512 376.9 394.3 (0.96x) 398.3 (0.95x) 377.8 (1.00x) 382.0 (0.99x)
1024 492.1 441.3 (1.12x) 444.4 (1.11x) 418.8 (1.18x) 426.9 (1.15x)
2048 899.1 626.2 (1.44x) 664.0 (1.35x) 572.4 (1.57x) 586.7 (1.53x)
4096 1591.8 1023.0 (1.56x) 1036.2 (1.54x) 941.1 (1.69x) 962.0 (1.65x)
8192 3144.2 1919.0 (1.64x) 1945.0 (1.62x) 1716.7 (1.83x) 1727.9 (1.82x)

Across the SM100 measurements, the crossover point sits between roughly 512 and 1024 tokens per rank. Below that region, deep_gemm_mega usually wins or stays at parity; above it, the CuteDSL variants pull ahead, and the quantized combine-wire variants extend the lead further at larger token counts.

1x4 SM103 — DeepSeek-V4-Flash geometry

tok/rank dg nvfp4 bf16 +ikr +combine_nvfp4 +combine_mxfp8
512 197.5 231.1 (0.85x) 229.3 (0.86x) 216.1 (0.91x) 220.2 (0.90x)
1024 229.4 259.9 (0.88x) 260.1 (0.88x) 233.4 (0.98x) 241.7 (0.95x)
2048 364.5 331.5 (1.10x) 334.8 (1.09x) 275.5 (1.32x) 291.8 (1.25x)
4096 610.3 533.5 (1.14x) 531.4 (1.15x) 398.3 (1.53x) 442.3 (1.38x)
8192 1123.3 988.2 (1.14x) 976.9 (1.15x) 698.0 (1.61x) 803.8 (1.40x)

On 1x4 SM103, the same pattern holds but the crossover moves up: because EP4 assigns twice as many experts per rank as EP8, the crossover generally shifts to roughly 1024 to 2048 tokens per rank. That is why the decode-1k gain is smaller on SM103 end-to-end, while the larger prefill-heavy cells still benefit more clearly.

Split path and MXFP8, BF16 microbenchmarks (1x8 SM100)

Mega-path columns below are e2e_pipelined p50 microseconds. Split-path columns below are nccl_ep HT barrier-cold e2e p50 microseconds.

DeepSeek-V4-Flash

tok/rank dg nvfp4 bf16 mxfp8 bf16 split fp4t split fp4c split w4a8
8 108.5 121.7 (0.89x) 171.1 (0.63x) 336.9 (0.32x) 1642.2 832.5 928.4
64 125.4 134.1 (0.94x) 197.7 (0.63x) 392.2 (0.32x) 1922.1 884.8 968.4
512 157.8 191.4 (0.82x) 263.2 (0.60x) 433.7 (0.36x) 1643.1 1158.6 1410.0
2048 383.0 335.5 (1.14x) 437.1 (0.88x) 822.2 (0.47x) 4811.5 3979.9 4949.4
8192 1324.1 1101.9 (1.20x) 1364.0 (0.97x) 2705.9 (0.49x) 11777.9 9423.6 13437.1

DeepSeek-V4-Pro

tok/rank dg nvfp4 bf16 mxfp8 bf16 split fp4t split fp4c split w4a8
8 260.2 259.1 (1.00x) 467.9 (0.56x) 1032.7 (0.25x) 2636.0 1741.6 1424.5
64 329.7 334.8 (0.98x) 646.1 (0.51x) 1449.0 (0.23x) 2478.1 1631.9 1716.3
512 374.8 394.3 (0.95x) 732.3 (0.51x) 1580.0 (0.24x) 2526.9 1838.3 2326.0
2048 884.3 618.9 (1.43x) 1166.4 (0.76x) 2434.1 (0.36x) 7631.7 6317.2 8096.2
8192 3176.0 1890.2 (1.68x) 3552.3 (0.89x) 7117.0 (0.45x) 22681.1 19984.6 27969.3

NCCL_EP vs NIXL_EP transport comparison

Identity-kernel comm-only comparison from the latest split-path measurements; values are barrier-cold e2e p50 microseconds.

tok/rank Flash nccl_ep HT Flash nccl_ep LL Flash nixl_ep LL Pro nccl_ep HT Pro nccl_ep LL Pro nixl_ep LL
8 315.9 113.0 127.8 229.1 124.4 82.7
64 315.2 144.2 87.2 799.2 599.4 98.4
512 315.9 186.9 165.0 367.3 239.0 258.6
2048 1973.5 413.3 — 2193.3 640.6 —
8192 3154.8 1372.8 — 3953.8 2355.0 —
  • fp4t = split NVFP4 TRT-LLM path.
  • fp4c = split NVFP4 CuteDSL path.
  • w4a8 = split W4A8 CuteDSL path.
  • nixl_ep LL is — at 2048 and 8192 tok/rank because max_tokens_per_rank is capped at 1024.

Accuracy gate

setup model native fi_dg fi_cutedsl delta
1x8 SM100 Flash 0.965 0.965 0.965 +0.000
1x8 SM100 Pro 0.880 0.880 0.890 +0.010
1x4 SM103 Flash 0.965 0.965 0.975 +0.010

Because fi_cutedsl runs an NVFP4-cast checkpoint rather than the MX checkpoint used by native and fi_dg, the throughput comparison needs to be considered alongside model quality. On these 200-question GSM8K runs, the measured deltas are zero to one percentage point. This small evaluation is a quality check for the reported configurations, not evidence of equivalent accuracy across tasks.

Architecture support

architecture status
SM100/SM103 Implemented
SM90 Implemented
SM120 kernel Implemented
SM107 Pending

Takeaways

The main conclusions are:

  • fi_dg stays near parity with native, so the FlashInfer moe_ep integration adds effectively no measurable overhead in the tested EP8 configuration.
  • The throughput gain is measured for the CuteDSL NVFP4 configuration, including its different checkpoint precision.
  • The gain appears consistently in both kernel and end-to-end measurements.
  • The gain increases with larger token counts and larger model geometry.
  • The reported GSM8K check accompanies the cross-checkpoint comparison; broader quality evaluation remains future work.

Future work

The following items remain active development directions rather than committed milestones:

  • Backpropagation support in collaboration with cudnn-frontend.
  • Broader model-level evaluation of the low-precision backends.

Part of FlashInfer v0.7. See the release highlights for upgrade details.