MultimodalFlow
中文
← Back to Blog

Jetson AGX Thor LLM Benchmark (2026): Qwen3.6-35B and Qwen2.5-1.5B Real-Hardware Results

JetsonLLMbenchmarkedge inferenceQwenSGLangllama.cppThor

All numbers in this post come from live hardware testing on a Jetson AGX Thor Developer Kit. No synthetic data, no manufacturer specs — just curl and llama-bench against real running models.

Test Environment

| Device | NVIDIA Jetson AGX Thor Developer Kit | |---|---| | CUDA Version | 13.0 | | Compute Capability | 11.0 | | Unified VRAM | 125,771 MiB (~123 GB) | | Total System RAM | 122 GB | | JetPack / Kernel | 6.8.12-tegra | | Storage | 936 GB NVMe | | GPU Temperature (idle) | 59°C |

Model 1: Qwen3.6-35B-A3B-FP8 via SGLang

Setup

The 35B model runs as a persistent SGLang server:

python3 -m sglang.launch_server \
  --model-path /models/Qwen3.6-35B-A3B-FP8 \
  --host 0.0.0.0 \
  --port 8080 \
  --served-model-name qwen3.6 \
  --trust-remote-code

The FP8-quantized variant is used. With 123 GB of unified memory, the full model loads without offloading — system memory shows 101 GB used with the server running.

Benchmark Results

Generation Speed (output tokens/second, 3-run average):

| Test | Prompt Tokens | Output Tokens | Time | Speed | |---|---|---|---|---| | Short prompt | 24 | 200 | 13.69s | 14.6 t/s | | Long prompt | 268 | 300 | 20.37s | 14.7 t/s |

Time to First Token (TTFT, streaming, 3-run average):

| Run | TTFT | |---|---| | Cold (first request) | 0.282s | | Warm (2nd request) | 0.101s | | Warm (3rd request) | 0.101s | | Average | 0.161s |

Memory Footprint:

| Metric | Value | |---|---| | System RAM with server idle | ~65 GB | | System RAM with model loaded | ~101 GB | | Model memory footprint (FP8, 35B) | ~36 GB | | Remaining available | ~21 GB |

What 14.6 t/s Means in Practice

Human reading speed is approximately 3–5 tokens per second. At 14.6 t/s, Qwen3.6-35B generates text roughly 3× faster than a person can read — making it genuinely comfortable for interactive chat, copilot tools, and real-time agent workflows.

For reference: cloud GPT-4o typically delivers 40–80 t/s, but requires internet connectivity, sends data off-device, and costs per token. At 14.6 t/s, the Thor trades some speed for complete local execution.

Model 2: Qwen2.5-1.5B Q4_K_M via llama.cpp (CUDA)

Setup

LD_LIBRARY_PATH=build/bin ./build/bin/llama-bench \
  -m /models/qwen2.5-1.5b/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 -fa 1 \
  -p 128,512 -n 128,256 \
  -r 3

All layers offloaded to GPU (-ngl 999), Flash Attention enabled (-fa 1).

Benchmark Results

| Model | Size | Backend | Flash Attn | Test | Speed (t/s) | |---|---|---|---|---|---| | Qwen2.5-1.5B Q4_K_M | 1.04 GiB | CUDA | ✓ | Prefill 128t | 3,639.6 ± 403.6 | | Qwen2.5-1.5B Q4_K_M | 1.04 GiB | CUDA | ✓ | Prefill 512t | 4,298.3 ± 158.5 | | Qwen2.5-1.5B Q4_K_M | 1.04 GiB | CUDA | ✓ | Generate 128t | 106.8 ± 6.4 | | Qwen2.5-1.5B Q4_K_M | 1.04 GiB | CUDA | ✓ | Generate 256t | 112.8 ± 0.1 |

Analysis

At 107–113 t/s generation speed, the 1.5B model runs approximately 22–30× faster than human reading speed. This makes it suitable for:

  • Real-time voice-to-text pipelines where transcription must keep pace with speech
  • High-throughput classification or extraction tasks
  • Multi-turn chat where latency should be invisible to the user

The prefill speed of 4,298 t/s for a 512-token context means even long prompts process in under 120ms — negligible for most applications.

Thor vs Orin: Capability Comparison

The test lab also runs a Jetson AGX Orin Developer Kit (100.97.175.73). Hardware comparison:

| Spec | Thor | Orin | |---|---|---| | CUDA Version | 13.0 | 12.6 | | Compute Capability | 11.0 | 8.7 | | Unified VRAM | 123 GB | 61 GB | | System RAM | 122 GB | 61 GB | | Max Model Size (FP16) | ~60B params | ~30B params | | Max Model Size (Q4) | ~230B params | ~115B params |

The Thor's 2× memory advantage is the defining difference. The Orin cannot run a 35B FP8 model without offloading; the Thor does it with room to spare.

For smaller models (up to 13B), the Orin remains a strong and cost-effective choice. The Qwen2.5-7B at Q4_K_M fits comfortably in the Orin's 61 GB unified memory and delivers approximately 28–35 t/s generation speed.

Key Takeaways

For edge AI deployment decisions:

  1. 35B+ models are now viable on-device — The Thor's 123 GB unified memory puts full-size reasoning models within reach for local deployment without cloud dependency.

  2. FP8 quantization is the sweet spot — The Qwen3.6-35B-A3B-FP8 variant delivers good generation quality at 14.6 t/s, using ~36 GB — about half what the BF16 version would require.

  3. Small models are extremely fast — The 1.5B model at 107+ t/s is fast enough for real-time applications where a lightweight model's capability is sufficient.

  4. TTFT under 0.2s for 35B — The 0.16s average TTFT for a 35B model is genuinely impressive for edge hardware. Users will not perceive any lag before the first token appears.

  5. SGLang is production-ready on Tegra — The SGLang server has been running continuously since May 25 with 9,781 CPU-hours of scheduler time, indicating solid stability.

Reproduce These Results

# Qwen2.5-1.5B via llama.cpp
cd ~/kwkthor/llama.cpp
LD_LIBRARY_PATH=build/bin build/bin/llama-bench \
  -m /home/nvidia/models/qwen2.5-1.5b/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 -fa 1 -p 128,512 -n 128,256 -r 3

# Qwen3.6-35B via SGLang API
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.6","messages":[{"role":"user","content":"Hello"}],"max_tokens":100}'

These benchmarks were collected on June 1, 2026 from live hardware. Results may vary with different JetPack versions, SGLang versions, or thermal conditions.