vLLM/Recipes
Search
Previewof vllm-project/vllm#64180 at 3f7318b.
MiniMax

MiniMaxAI/MiniMax-M3.1-Flash

MiniMax M3.1-Flash vision-language MoE (~428B total / 23B active) with MSA sparse attention throughout and a built-in DSpark draft; quantization-aware trained, MXFP8 dense layers and NVFP4 experts. Served TP4 on NVIDIA Blackwell or AMD MI355X, prefill/decode disaggregated as TP4 + TP4 on MI355X or DEP2 + DEP4 on GB200/GB300, and TP8 on NVIDIA Hopper.

MSA sparse attention, 1M context, DSpark speculative decoding built into the checkpoint

View on HuggingFace
moe428B / 23B1,048,576 ctxvLLM 0.33.0+textmultimodal

Overview

MiniMax-M3.1-Flash is a vision-language MoE model from MiniMax that keeps the MoE architecture of MiniMax-M3: about 428B total / 23B active parameters and a 1M-token context, with MSA sparse attention in every layer.

  • Quantization-aware training — the checkpoint stores MXFP8 dense layers and NVFP4 experts.
  • Built-in DSpark draft — the dspark/ directory of the checkpoint; --speculative-config needs no draft model path.
  • Native multimodal — image and video inputs.

Prerequisites

  • AMD: MI355X (gfx950), ROCm 7.2+, TP4.
  • NVIDIA Blackwell: GB200, GB300, B200 or B300, TP4; NVFP4 MoE runs natively on the FP4 tensor cores.
  • NVIDIA Hopper: H100 or H200, TP8 on a full node, with the FP8 variant: FP8 KV and indexer caches, NVFP4 MoE on Marlin.
  • --block-size 128 is mandatory (MSA sparse/index cache).

Launching the Server

Single node (TP4, AMD MI355X)

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  -tp 4 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
  --quantization-config.moe mxfp8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
  --disable-hybrid-kv-cache-manager
  • --kv-cache-dtype nvfp4 and indexer_kv_dtype: nvfp4 store the KV cache and the sparse-attention indexer's K cache in NVFP4.
  • indexer_decode_cp splits the indexer's decode scoring across the TP ranks by context instead of by index head.
  • --quantization-config.moe mxfp8 re-quantizes the NVFP4 experts to MXFP8 at load for the AITER MoE.

DBO and SP prefill (TP4, AMD MI355X)

Two opt-in switches make long prefill steps faster on the single-node MI355X engine (the DBO / SP prefill feature in the command builder):

  • DBO (dual-batch overlap): --enable-dbo splits a prefill step into two microbatches and runs one microbatch's all-reduce under the other's compute. --dbo-decode-token-threshold 32768 keeps decode steps whole. DBO needs VLLM_USE_V2_MODEL_RUNNER=1: without it, --enable-dbo selects the V1 runner, which refuses the DSpark draft.
  • SP (sequence-parallel prefill): VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1 keeps the residual stream sharded by tokens on steps of 4096+ tokens, so each layer's two all-reduces become an all-to-all and a reduce-scatter. It keeps a full o_proj copy per GPU, about 3.1 GB less KV cache.
  • SP + DBO: set both, with --dbo-prefill-token-threshold 16384, so each DBO microbatch has 8192+ tokens and stays on the SP path.

On the MXFP8 MoE, one 32K-token prefill takes 1035 ms plain, 655 ms with DBO and 484 ms with SP + DBO. The server log confirms SP with MiniMax-M3 sequence-parallel prefill (from 4096 tokens, DBO microbatches from 8192): on.

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ROCM_MINIMAX_M3_SP_PREFILL=1 \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  -tp 4 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
  --quantization-config.moe mxfp8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
  --enable-dbo \
  --dbo-prefill-token-threshold 16384 \
  --dbo-decode-token-threshold 32768 \
  --disable-hybrid-kv-cache-manager

Single node (TP4, NVIDIA GB200)

VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  -tp 4 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
  --max-num-seqs 256 \
  --disable-hybrid-kv-cache-manager
  • Blackwell runs the NVFP4 experts natively, so no --quantization-config.moe override is needed.
  • use_prefill_query_quantization quantizes the prefill query on the MSA indexer path (Q8KV4).

Single node (TP8, NVIDIA H100 / H200)

vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  --tensor-parallel-size 8 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype fp8 \
  --attention-config '{"indexer_kv_dtype": "fp8"}' \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
  --max-model-len 262144 \
  --max-num-seqs 80 \
  --limit-mm-per-prompt '{"video": 0}' \
  --gpu-memory-utilization 0.9 \
  --no-enable-log-requests \
  --disable-hybrid-kv-cache-manager \
  --moe-backend marlin \
  --enable-auto-tool-choice \
  --tool-call-parser minimax_m3
  • Hopper has no FP4 tensor cores: the KV cache and the indexer K cache use FP8, and the NVFP4 experts run on the Marlin MoE kernel.
  • --limit-mm-per-prompt '{"video": 0}' turns off video inputs; images still work.
  • DSpark speculative decoding is not enabled on Hopper.

Prefill/decode disaggregation (1P1D, TP4 + TP4, AMD MI355X)

Prefill node:

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST=<P_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  -tp 4 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4"}' \
  --quantization-config.moe mxfp8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
  --disable-hybrid-kv-cache-manager \
  --max-num-batched-tokens 32768 \
  --port 8100 \
  --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[
    {"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"num_threads":4}},
    {"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":601295421440,"cpu_hugepage_block_size":"1GB","lazy_offload":false}}]}}'

Decode node:

VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 \
VLLM_KV_CACHE_LAYOUT=BLHNC VLLM_NIXL_SIDE_CHANNEL_HOST=<D_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  -tp 4 \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "indexer_decode_cp": true}' \
  --quantization-config.moe mxfp8 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic"}' \
  --disable-hybrid-kv-cache-manager \
  --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"num_threads":4}}'

  • <P_IP> and <D_IP> are each node's own address, reachable from the other node (the NIXL side channel).

  • VLLM_KV_CACHE_LAYOUT=BLHNC goes on both nodes. It keeps all layers of a block contiguous, so NIXL reads one region per block instead of one per layer.

  • The prefill node never decodes, so it leaves indexer_decode_cp off.

  • The prefill node also offloads KV to host memory (SimpleCPUOffloadConnector): 560 GiB per rank in 1 GiB HugeTLB pages, so 2240 pages for the four ranks. Reserve them on the host before the start (and allow them in the container's hugetlb limit), or the server fails at startup with Cannot map 560.00 GiB of 1024 MiB HugeTLB pages:

    echo 2240 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Prefill/decode disaggregation (1P1D, DEP2 + DEP4, NVIDIA GB200 / GB300)

A DEP2 prefill on one 4-GPU tray feeds a DEP4 decode on a second tray over NIXL. Both roles run attention data parallel with the NVFP4 experts on the DeepGEMM MegaMoE kernel over expert parallelism, the V2 model runner, NUMA binding, and a seven-token DSpark draft on the FlashInfer PrimiTS (FLASHINFER_TS) backend. The settings were tuned on GB300; GB200 uses the same commands.

Prefill node:

VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \
VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST=<P_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  --data-parallel-size 2 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --numa-bind \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
  --max-num-seqs 64 \
  --max-num-batched-tokens 32768 \
  --long-prefill-token-threshold 8192 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --disable-hybrid-kv-cache-manager \
  --port 8100 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}'

Decode node:

VLLM_FLOAT32_MATMUL_PRECISION=high VLLM_USE_V2_MODEL_RUNNER=1 \
VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_NIXL_SIDE_CHANNEL_HOST=<D_IP> \
vllm serve MiniMaxAI/MiniMax-M3.1-Flash \
  --data-parallel-size 4 \
  --enable-expert-parallel \
  --moe-backend deep_gemm_mega_moe \
  --numa-bind \
  --reasoning-parser minimax_m3 \
  --block-size 128 \
  --kv-cache-dtype nvfp4 \
  --attention-config '{"indexer_kv_dtype": "nvfp4", "use_prefill_query_quantization": true}' \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", "max_cudagraph_capture_size": 224}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"attention_backend":"FLASHINFER_TS","kv_cache_dtype":"fp8","draft_sample_method":"probabilistic","rejection_sample_method":"block"}' \
  --max-num-seqs 28 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.95 \
  --watermark 0.05 \
  --no-enable-prefix-caching \
  --disable-hybrid-kv-cache-manager \
  --port 8200 \
  --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_connector_extra_config":{"engine_ttl":0}}'
  • VLLM_ENGINE_READY_TIMEOUT_S=3600: with data parallel, each API server waits this long for its engines to start (default 600 s). A cold prefill start (JIT builds, CUDA graph capture, FlashInfer autotune) takes longer than 10 minutes.
  • --max-num-seqs is per DP rank. The decode CUDA graphs cover 28 requests of 1 + 7 tokens (224).
  • --watermark 0.05 keeps 5% of the decode KV cache free when admitting requests, so extra requests wait in the queue instead of preempting running ones. On GB300 (about 4.5M KV tokens per decode rank) at a 200K-token mean context, expect about 19-21 running requests per rank.
  • Keep the decode --max-num-batched-tokens at 8192: a preempted request recomputes its context in chunks of that size, and every DP rank runs eager steps until it finishes.
  • engine_ttl: 0 turns off NIXL's cleanup of idle remote engines; with the default one-hour TTL, a decode crashed an hour after its prefill went away.
  • Prefix caching stays on the prefill node only. For multi-turn traffic, the prefill node can also offload KV to host memory, 180 GiB per rank: --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_producer","kv_connector_extra_config":{"engine_ttl":0}},{"kv_connector":"SimpleCPUOffloadConnector","kv_role":"kv_both","kv_connector_extra_config":{"cpu_bytes_to_use_per_rank":193273528320,"lazy_offload":true}}]}}'.

For a quick test, put the toy NIXL proxy from the vLLM repository in front of the two nodes and send requests to port 8000:

python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py --port 8000 \
  --prefiller-host <P_IP> --prefiller-port 8100 \
  --decoder-host <D_IP> --decoder-port 8200

Client Usage

Recommended sampling parameters (from the model card): temperature = 1.0, top_p = 0.95. Set the reasoning effort per request with chat_template_kwargs: {"reasoning_effort": "<level>"}: low, medium, high, xhigh or max.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMaxAI/MiniMax-M3.1-Flash",
    "temperature": 1.0,
    "top_p": 0.95,
    "messages": [
      {"role": "user", "content": "Explain MSA sparse attention in 3 bullets."}
    ]
  }'

Benchmarking

vllm bench serve \
  --backend vllm \
  --model MiniMaxAI/MiniMax-M3.1-Flash \
  --endpoint /v1/completions \
  --dataset-name random \
  --random-input 8192 \
  --random-output 1024 \
  --max-concurrency 32 \
  --num-prompts 320

Troubleshooting

  • --block-size mismatch. MSA's sparse block size is 128; the vLLM KV cache block size must match (otherwise No common block size for 16).
  • Parsers. --tool-call-parser and --reasoning-parser both use minimax_m3.
  • HugeTLB pages (PD prefill node). See the AMD PD section above.

References