MultimodalFlow
中文
← Back to Blog

DeepSeek V4 Flash NVFP4 on 2×Jetson Thor, Debug Log 1: SGLang Reaches DeepGEMM

DeepSeek-V4DeepSeek-V4-FlashNVFP4Jetson ThorSGLangvLLMQSFP28Edge AI

This is a debug log, not a success report.

The target was simple: run nvidia/DeepSeek-V4-Flash-NVFP4 across two Jetson Thor systems connected by QSFP28. This first round focused on the SGLang path. The vLLM / DSpark path is next.

ItemCurrent result
Modelnvidia/DeepSeek-V4-Flash-NVFP4
RuntimeSGLang nightly lmsysorg/sglang:nightly-dev-cu13-20260812-c7c03ec5
Hardware2× Jetson Thor
NetworkQSFP28, 4×10GbE; aggregate iperf around 35.6 Gbit/s
ParallelismTP=1 + PP=2
Current stateServing did not start successfully
ProgressDistributed init, weight loading, DSV4 memory pool, and FlashInfer autotune all completed
Final blockerdeepgemm/csrc/apis/hyperconnection.hpp:59: Unsupported architecture

Short version: the system, network, containers, and checkpoint loading are mostly working. The remaining SGLang blocker is DeepGEMM HyperConnection / MHC kernel support for Thor sm110.

Hardware and Network

The two nodes:

NameTailscaleQSFP28 side
thor100.98.202.31.1 side
thor-jd100.98.98.105.2 side

The four QSFP28 mgbe subnets:

Interfacethorthor-jd
mgbe0_0192.168.100.1/24192.168.100.2/24
mgbe1_0192.168.101.1/24192.168.101.2/24
mgbe2_0192.168.102.1/24192.168.102.2/24
mgbe3_0192.168.103.1/24192.168.103.2/24

This SGLang run used mgbe1_0:

NCCL_SOCKET_IFNAME=mgbe1_0
GLOO_SOCKET_IFNAME=mgbe1_0
NCCL_IB_DISABLE=1
NCCL_NET=Socket

The QSFP28 aggregate iperf result was:

Linkiperf3 result
mgbe0_08.87 Gbit/s
mgbe1_08.87 Gbit/s
mgbe2_08.95 Gbit/s
mgbe3_08.91 Gbit/s
Total35.61 Gbit/s

The setup notes are in docs/thor-qsfp28-40g-setup.md.

Weights and Image

The checkpoint was first downloaded to a Ubuntu relay machine, then copied to both Thor nodes:

thor:    /home/nvidia/models/DeepSeek-V4-Flash-NVFP4
thor-jd: /home/nvidia/models/DeepSeek-V4-Flash-NVFP4

The model directory is roughly 157GB and contains 46 safetensors shards.

SGLang image:

lmsysorg/sglang:nightly-dev-cu13-20260812-c7c03ec5

Key versions inside the container:

torch:        2.13.0+cu130
CUDA:         13.0
transformers: 5.12.1
sglang:       0.0.0.dev1+gc7c03ec53
GPU:          NVIDIA Thor, capability (11, 0)

This nightly recognizes deepseek_v4. Older NGC sglang:26.04-py3 and vllm:26.04-py3 images were too old for this checkpoint.

Docker Runtime Fix

On thor-jd, Docker initially failed with:

failed to inject CDI devices:
unresolvable CDI devices runtime.nvidia.com/gpu=all

On this Jetson / L4T setup, NVML-based CDI generation also failed:

Unable to determine the device handle for GPU0: 0000:01:00.0: Unknown Error

The working fix was to force the NVIDIA container runtime into CSV mode:

sudo cp /etc/nvidia-container-runtime/config.toml /etc/nvidia-container-runtime/config.toml.bak
sudo sed -i 's/mode = "auto"/mode = "csv"/' /etc/nvidia-container-runtime/config.toml
sudo systemctl restart docker

After that, the SGLang image saw CUDA correctly:

True
NVIDIA Thor
(11, 0)

Why PP Instead of Cross-Node TP

Cross-node TP=2 was not a good fit for this Ethernet QSFP28 setup. SGLang logged:

CustomAllreduce is disabled because this process group spans across nodes

For DeepSeek V4 with MoE, MLA, and FP4/NVFP4 kernels, tensor parallelism across Ethernet is expensive. This run switched to:

TP=1 + PP=2

Each Thor owns a pipeline stage instead of splitting tensor operations across the network.

What Worked

Distributed initialization completed:

[PP0] Init torch distributed ends. elapsed=14.17 s
[PP1] Init torch distributed ends. elapsed=28.51 s

Weight loading completed:

[PP0] Load weight end. elapsed=110.23 s
[PP1] Load weight end. elapsed=86.11 s

SGLang detected:

DeepseekV4ForCausalLM
hybrid FP8+NVFP4 checkpoint
NVFP4 MoE group_size=16

The DSV4 memory pool was created:

[PP0] DSV4 pool PP slice: rank=0 layers=[0,21) local=21/44
[PP1] DSV4 pool PP slice: rank=1 layers=[21,43) local=22/44

And FlashInfer autotune started for sm110:

/root/.cache/sglang/flashinfer/autotune/0.6.15.post1/sm110/...

Final Failure

Both nodes failed during warmup forward:

Scheduler hit an exception
self.warmup()
RuntimeError: Assertion error (/deepgemm/csrc/apis/hyperconnection.hpp:59):
Unsupported architecture

Disabling TileLang MHC pre/post and prewarm was not enough:

SGLANG_OPT_USE_TILELANG_MHC_PRE=0
SGLANG_OPT_USE_TILELANG_MHC_POST=0
SGLANG_DSV4_MHC_PREWARM=0

The runtime path still reached DeepGEMM HyperConnection / MHC, which currently does not support Thor sm110.

Current Engineering Read

This test showed:

  1. The checkpoint can be read on both Thor nodes.
  2. SGLang nightly supports the deepseek_v4 model class and hybrid FP8+NVFP4 metadata.
  3. QSFP28 is good enough for torch distributed rendezvous in TP=1 + PP=2.
  4. Weight loading and DSV4 memory pool allocation work.
  5. The current SGLang blocker is kernel architecture support, not disk, networking, or model recognition.

The next options are:

PathCostComment
Patch SGLang to fully bypass DeepSeek V4 MHC at runtimeMediumBest chance to get first tokens, likely slower
Add / rebuild DeepGEMM HyperConnection support for sm110HighKernel-level work
Try vLLM / DSparkMediumAlready starts on Thor and sees the GPU

Next: vLLM / DSpark

The dual DGX Spark reference uses:

ghcr.io/anemll/dspark-vllm-gx10:0.1.1
vLLM 0.25.2
DeepSeek-V4-Flash-0731
TP=2
nvfp4_ds_mla KV cache

That image has already been loaded on thor and the single-node probe works:

Linux-6.8.12-tegra-aarch64
torch 2.11.0+cu130
CUDA 13.0
cuda available: True
GPU: NVIDIA Thor
capability: (11, 0)
vLLM: 0.25.2.dev0
transformers: 5.13.1

That does not prove DeepSeek V4 will run, because the image is clearly tuned for DGX Spark / GB10 / sm_121a, while Thor is sm110. But it is the next reasonable path to test.

References:

Test date: August 13, 2026. This is debug log 1 and does not claim successful DeepSeek V4 Flash NVFP4 inference on 2×Jetson Thor.