120B-Class Models on a 128 GB Thor: Qwen3.5-122B vs Flash-Next NVFP4
On November 4–5, 2026, I deployed nvidia/Qwen3.5-122B-A10B-NVFP4 and nvidia/Qwen3.8-Flash-Next-NVFP4 on the same Jetson AGX Thor. Both served text with a 65,536-token context limit, FP8 KV cache and at most two active sequences. Observed single-stream generation was approximately 16 and 22 tok/s, respectively.
These are small-sample deployment observations. The vLLM versions differed and cache states were not fully isolated. They establish feasibility for these configurations rather than a controlled architecture ranking.
Environment and evidence
| Item | Configuration |
|---|---|
| Device | Jetson AGX Thor, marketed 128 GB unified memory |
| Compute capability | sm_110 |
| System | JetPack / L4T R39.2, MAXN |
| Host CUDA driver API | 13.2 |
| Qwen3.5 runtime | vllm/vllm-openai:v0.28.0 |
| Flash-Next runtime | Official nightly-aarch64, 0.33.0rc1.dev806+ga2bcb5657 |
| Serving | 64K limit, FP8 KV, two active sequences, prefix caching |
| Modality | Language-model-only; image/video inputs not evaluated |
The host CUDA driver API and container CUDA runtime are separate fields: the 122B deployment README records PyTorch 2.13.0 and CUDA 13.0 inside its container.
I checked the saved Thor README and launch scripts over SSH on November 10. The downloadable deployment summary identifies its sources; it is a transcription of observations, not per-request raw data. The companion article covers the small-model validation and three API replay demos.
Qwen3.5-122B-A10B
The NVIDIA model card describes 122B total parameters and 10B active parameters. The tested revision was 98915d837c4e7c87ac8296d02e89de19b3207e6d.
| Observation | Value |
|---|---|
| Weight files | 77.79 GiB |
| Runtime model allocation | 72.37 GiB |
| KV cache at the 64K configuration | 12.02 GiB |
| Reported KV token capacity | 747,706 |
| Single-stream generation | Approx. 16 tok/s |
| Two-stream aggregate output | Approx. 27.8–31.9 tok/s |
| Short-input TTFT | Approx. 0.4 s |
The two concurrency numbers came from separate rounds: 27.79 tok/s in the first table and 31.9 tok/s in a later comparison. The reported KV capacity does not establish a supported service concurrency above the configured limit of two.
Time to first token rose to 7.6 seconds at approximately 14K input tokens, 19.3 seconds at 30K and 50.5 seconds at 60K. A large context capacity still needs a latency budget.
Flash-Next: making room with PLE disk offload
The lab notes describe the tested checkpoint as approximately 125B total / 6B active parameters, with MoE, Gated DeltaNet, QSA sparse attention and PLE n-gram embeddings. This parameter description applies to the checkpoint in this experiment, not every similarly named variant.
Revision: fc694b54fb0174e0913e6adf86691ef85a4ead47. The runtime was pinned to:
vllm/vllm-openai@sha256:cca10003856866925653fb4ec45309df385dc15f802dbd47c289eb9c5720f197
The important settings were:
--engram-config '{"disk_offload": true}' \
--kv-cache-dtype fp8 \
--max-model-len 65536 \
--max-num-seqs 2 \
--mamba-cache-mode align \
--enable-prefix-caching \
--gpu-memory-utilization 0.75
About 54 GB of PLE tables remained file-backed instead of being allocated as a complete resident table. Runtime model allocation was 72.89 GiB. This is not total system memory: KV cache, buffers, the OS and file caching still require space.
The vLLM Engram documentation explains that this disk path reads the needed table rows from checkpoint files; recently read data can be served by the OS page cache. Disk offload therefore avoids whole-table residency but does not eliminate RAM usage or cold-storage latency.
Observed comparison
| Metric | Qwen3.5-122B | Flash-Next |
|---|---|---|
| Single-stream generation | Approx. 16 tok/s | Approx. 22 tok/s |
| Two-stream aggregate | Approx. 27.8–31.9 tok/s | Approx. 37.9 tok/s |
| 30K-input TTFT | 19.3 s | 14.0 s |
| 60K-input TTFT | 50.5 s | 27.9 s |
These observations imply roughly 38% higher single-stream throughput and 27% / 45% lower TTFT at 30K / 60K input. Using the later 31.9 tok/s baseline gives approximately 19% higher two-stream throughput; using the earlier round gives a different percentage.
Hybrid attention and fewer active parameters are plausible contributors. However, the model, runtime, kernels and cache conditions changed together. This experiment does not isolate the benefit of any one component.
For one prompt, “Explain MoE in three sentences,” the 122B model produced 2,034 tokens in 125 seconds; Flash-Next produced 156 tokens in 7.5 seconds. Completion was about 17 times faster, largely alongside a much shorter response. That is not a 17-fold decoding improvement or evidence of universally better reasoning efficiency or quality.
The FP4-Local / b12x failure
The nvidia/Qwen3.8-Flash-Next-FP4-Local attempt used eugr/spark-vllm-b12x:nightly-20261102 with CUTE_DSL_ARCH=sm_110a. Initialization failed:
ValueError: Failed to find a kernel that can implement the NVFP4 linear layer.
B12xNvFp4LinearKernel: B12X NVFP4 kernels require a Blackwell 12x device
Changing an architecture variable cannot supply an sm_110 implementation missing from that kernel. Check compatibility at the kernel level before transferring a Spark recipe to Thor. This failure applies to the tested image and configuration; the standard NVFP4 path above succeeded.
Replaying the deployment
The adapted launch script pins the model revisions and listens on loopback. Download the selected revision to the Thor Hugging Face cache first: the script loads offline. Run one large model at a time and ensure the port and container name are available.
# Choose one, on the Thor with model cache and NVIDIA runtime prepared
bash serve.sh flash-next
# Alternatively:
bash serve.sh qwen35
curl http://127.0.0.1:8000/v1/models
docker logs --tail 100 qwen38-flash-next
No MTP performance or sustained-service quality measurements were collected. A later test lost access to the machine while Flash-Next was resident and only about 18 GB was available. That warrants memory-headroom checks, but the saved evidence does not prove a unique kernel, memory or image-pull root cause.
At the November 10 inspection, the Thor had rebooted and no large-model service was running. The examples are replay instructions, not a public live endpoint. The next useful experiment is a larger request set with controlled cache state, matched runtimes, task quality checks and a sustained run.