Are NVIDIA Jetson AGX Thor LLM Benchmarks Realistic? I Retested on a Live Deployment
NVIDIA now publishes Jetson AGX Thor LLM and VLM benchmark numbers for models such as Llama, Qwen, DeepSeek, and Qwen2.5-VL. The numbers look strong, and they are useful.
But they should not be treated as the speed your production application will automatically see.
The right interpretation is simple: official benchmarks are upper-bound references under controlled, optimized conditions; real edge deployments need workload-specific measurements.
To avoid keeping this purely theoretical, I connected to a live Jetson AGX Thor test system, referred to below as thor-test, and ran a lightweight benchmark against the existing SGLang service without stopping background services, clearing the desktop session, or pulling a new model.
What the Official Benchmark Measures
According to NVIDIA's Jetson Benchmarks page, the Jetson AGX Thor generative AI results were measured under roughly these conditions:
| Item | Official Condition |
|---|---|
| Device | NVIDIA Jetson AGX Thor Developer Kit |
| System | JetPack 7.0 |
| CUDA | 13.0 |
| TensorRT | 10.13 |
| Runtime | vLLM |
| Input / output length | ISL 2048 / OSL 128 |
| Concurrency | C=1 and C=8 |
Selected official LLM / VLM results:
| Type | Model | C=1 tokens/s | C=8 tokens/s |
|---|---|---|---|
| LLM | Llama 3.1 8B | 41.3 | 150.8 |
| LLM | Llama 3.3 70B | 4.7 | 12.6 |
| LLM | Qwen 3 30B-A3B | 61.0 | 226.4 |
| LLM | Qwen 3 32B | 13.19 | 79.1 |
| LLM | DeepSeek R1 7B | 41.32 | 304.8 |
| LLM | DeepSeek R1 32B | 13.31 | 82.6 |
| VLM | Qwen2.5-VL 3B | 71.7 | 356.86 |
| VLM | Qwen2.5-VL 7B | 45.0 | 252.0 |
| VLM | Llama 3.2 11B Vision | 26.31 | 69.63 |
Source: NVIDIA Jetson Benchmarks page, checked on 2026-07-27.
The key detail is not any single number. It is the benchmark setup: vLLM, fixed ISL/OSL, explicit concurrency, and NVIDIA's optimized software stack.
If your deployment uses llama.cpp, SGLang, Ollama, different quantization, longer context, or a device that is also running camera pipelines, ROS, web services, and Python automation, you should expect different results.
Why Real Deployments Often Look Slower
1. Tokens/s Is Not One Metric
LLM inference has at least two phases:
| Phase | Main Drivers | User-Visible Effect |
|---|---|---|
| Prefill | Input length, KV cache, batch size, attention implementation | Time to first token |
| Decode | Model size, quantization, memory bandwidth, scheduler behavior | Output speed |
The official table uses ISL 2048 / OSL 128. That is a clear workload, but real chat, agents, RAG, and visual QA do not always look like 2048 input tokens and 128 output tokens.
A RAG request may include 8K tokens of retrieved context. A robotics status summary may use only 200 input tokens. A coding agent may repeatedly append tool results to history. These workloads stress different parts of the system.
2. Total Throughput Is Not Single-User Latency
The official C=8 numbers are often much higher than C=1 because concurrent requests improve GPU utilization. For example, Qwen 3 30B-A3B is listed at 61 tokens/s for C=1 and 226.4 tokens/s for C=8.
That does not mean one user will see 226 tokens/s. It means aggregate throughput is higher across eight concurrent requests.
If you are building a handheld field assistant, single-request latency matters more.
If you are summarizing events from many factory cameras, total throughput may matter more.
3. Runtime Choice Changes Everything
The same model can behave very differently across inference engines:
| Engine | Strength | Common Cost |
|---|---|---|
| vLLM | High throughput, PagedAttention, mature concurrent serving | Heavier container and tuning path on edge devices |
| SGLang | Good for structured generation, agents, serving workflows | Model support can vary |
| llama.cpp | Mature GGUF ecosystem, simple deployment | Peak throughput is usually lower than specialized serving stacks |
| Ollama | Very easy validation path | Less benchmark control |
| TensorRT-LLM / Edge-LLM | Highest optimization potential on NVIDIA hardware | More export, build, and debugging complexity |
The official benchmark uses vLLM. Your GGUF result from llama.cpp should not be expected to match it automatically.
4. Edge Devices Rarely Run Only the Model
A real Jetson deployment may also run:
- camera capture and encoding
- ROS or Isaac ROS nodes
- object detection or VLM pipelines
- local web APIs
- logs, monitoring, remote access
- Python automation scripts
These consume CPU, memory bandwidth, GPU scheduling headroom, and thermal budget. A lab benchmark measures the model. A deployment measures the system.
Live Deployment State: thor-test
This was not a clean lab benchmark. It was a Thor system that had already been running services for two weeks:
| Item | State |
|---|---|
| Host | thor-test |
| OS | Ubuntu 24.04.3 LTS |
| L4T | R38.2.2 |
| Kernel | 6.8.12-tegra |
| Power mode | MAXN |
| Memory | 122 GiB total, about 61 GiB used before the test |
| Idle temperature | GPU around 58-60C, CPU / Tj around 60C |
Services already running during the test:
| Service | Notes |
|---|---|
| SGLang model service | Qwen2.5-7B-Instruct |
| Vector retrieval service | Used for local RAG / document retrieval |
| Text search service | Used for keyword search |
| Local model management service | Used for quick validation of other models |
| Desktop and remote management processes | Representative of a maintained edge node |
This is closer to a real edge AI node: the model server is not the only workload. The same box also handles retrieval, management, remote access, and application processes.
Retest Result: SGLang + Qwen2.5-7B-Instruct
The live model service:
Runtime: SGLang
Container image: nvcr.io/nvidia/sglang:26.04-py3
Model: Qwen2.5-7B-Instruct
Context length: 4096
mem-fraction-static: 0.3
API: OpenAI-compatible /v1/chat/completions
Single-Request Test
Each request used max_tokens=160 with non-streaming responses:
| Case | Prompt tokens | Completion tokens | Time | Completion tokens/s |
|---|---|---|---|---|
| Chinese edge AI Q&A | 46 | 136 | 8.180s | 16.63 |
| English summary | 44 | 65 | 3.958s | 16.42 |
| JSON output | 50 | 32 | 1.993s | 16.06 |
The result was consistent: with background services, desktop processes, retrieval components, and remote access running, Qwen2.5-7B-Instruct delivered about 16 completion tokens/s for single requests.
4-Concurrent-Request Test
I then sent four concurrent requests, each capped at 128 output tokens:
| Metric | Result |
|---|---|
| Concurrency | 4 |
| Total wall time | 8.959s |
| Total completion tokens | 505 |
| Aggregate completion tokens/s | 56.37 |
| Per-request latency | 8.534-8.957s |
This explains why official C=1 and C=8 tables need to be read separately: concurrency can raise total throughput substantially, but individual user latency does not improve by the same ratio.
How This Compares With the Official Benchmark
This retest should not be compared directly against NVIDIA's official Qwen 3 30B-A3B or Qwen 3 32B results because the conditions are different:
| Dimension | Official Benchmark | thor-test Retest |
|---|---|---|
| System | JetPack 7.0 | L4T R38.2.2 / Ubuntu 24.04.3 |
| Runtime | vLLM | SGLang |
| Model | Qwen 3 family and others | Qwen2.5-7B-Instruct |
| ISL / OSL | 2048 / 128 | Short prompts / up to 160 output tokens |
| Device state | Benchmark environment | Live services running |
| Background load | Controlled | Model serving, retrieval services, local model management, desktop and remote management processes |
So the right conclusion is not "official fast, real slow." It is:
In a live service environment, Thor can still provide stable local API serving for a 7B-class model: about 16 tokens/s for single requests and about 56 aggregate tokens/s at 4-way concurrency.
To reproduce NVIDIA's table, the next step would be a stricter apples-to-apples test with the same JetPack / CUDA / TensorRT / vLLM / model / ISL / OSL / concurrency setup.
Previous Larger-Model Result
In an earlier test, I ran Qwen3.6-35B-A3B-FP8 on Jetson AGX Thor with SGLang. The device was not in a clean benchmark state.
Observed result:
| Model | Runtime | Condition | Result |
|---|---|---|---|
| Qwen3.6-35B-A3B-FP8 | SGLang | Background load, 160-token responses | ~14 tokens/s |
This number should not be compared directly against NVIDIA's Qwen 3 30B-A3B vLLM result. The model, runtime, prompt length, and device load were different.
But it does show the more useful engineering point:
Thor has enough memory and compute headroom for mid-to-large local models. The actual user experience depends on runtime choice, concurrency strategy, and workload shape.
For a fair apples-to-apples comparison, I would need a clean test system, JetPack 7.0, the same model, vLLM, and the same ISL 2048 / OSL 128 workload.
How I Would Retest the Official Benchmark
If the goal is to see whether your Thor setup is close to NVIDIA's baseline, I would test it in this order.
Step 1: Record the System Snapshot
cat /etc/nv_tegra_release
uname -a
nvidia-smi || true
python3 -V
free -h
df -h
JetPack, CUDA, container, and runtime versions matter. Do not just write "Thor"; record the full software stack.
Step 2: Fix Power and Thermal Conditions
sudo nvpmodel -q
sudo jetson_clocks --show
tegrastats
Let the device reach a stable state before measurement. Record idle temperature and power. Thermal behavior directly affects sustained edge throughput.
Step 3: Reproduce the Official Workload First
Before testing your real application, establish a baseline close to the official setup:
| Parameter | Recommendation |
|---|---|
| ISL | 2048 |
| OSL | 128 |
| Concurrency | 1 and 8 |
| Runtime | vLLM |
| Metrics | prefill tokens/s, decode tokens/s, end-to-end latency, peak memory |
Only after this step can you tell whether your device and software stack have a baseline problem.
Step 4: Measure Your Actual Workload
For production, I would add at least three workload groups:
| Scenario | Input | Output | Focus |
|---|---|---|---|
| Interactive chat | 512-2K | 128-512 | Time to first token |
| RAG QA | 4K-16K | 256-1K | Long-context prefill |
| Agent / JSON | 1K-8K | Multi-turn short outputs | Format reliability and tool calls |
For VLMs, also record image resolution, number of images, vision encoder time, and total latency.
How to Read Thor's Official Numbers
I would treat the official benchmark as three signals.
Signal 1: Large Models Now Fit on Edge Hardware
Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 32B, and Qwen2.5-VL 7B all appear in the Thor benchmark table. That alone matters.
Thor's 128GB-class unified memory puts edge devices into a new range: local serving for tens-of-billions-parameter LLMs and VLMs.
Signal 2: MoE Models Are Especially Interesting on Edge
Qwen 3 30B-A3B is listed at 61 tokens/s for C=1 and 226.4 tokens/s for C=8. The key point is not simply that a "30B" model looks fast. It is that the model activates only part of its parameters per forward pass.
For edge deployment, MoE is attractive because it offers larger model capacity with lower active compute. The tradeoff is that routing, runtime support, and quantization need more care.
Signal 3: VLMs May Be Thor's Most Important Use Case
The Qwen2.5-VL and Llama 3.2 Vision results are more interesting for edge AI teams than pure text LLM numbers.
Thor deployments are rarely just chatbots. They are more likely to:
- read camera frames
- inspect gauges and panels
- explain anomaly screenshots
- combine voice and vision for field assistants
- generate action suggestions for robots or industrial systems
This is where MultimodalFlow should keep testing.
Practical Selection Guide
| Goal | Test First |
|---|---|
| Single-user local assistant | C=1, time to first token, decode tokens/s |
| Multi-client API service | C=4 / C=8 throughput, P95 latency |
| RAG knowledge base | Prefill time at 8K / 16K input |
| Industrial visual QA | VLM image encoding time + total response time |
| Robotics agent | Multi-turn stability, JSON / tool-call success rate |
| Offline field deployment | Thermal stability, restart behavior, logs, monitoring |
If you look at only one tokens/s number, it is easy to choose the wrong model.
If you split the workload, Thor's value becomes much clearer.
Conclusion
The Jetson AGX Thor official benchmark is not meaningless marketing. It is a useful upper-bound reference for what Thor can do under JetPack 7.0, CUDA 13.0, TensorRT 10.13, vLLM, and fixed ISL/OSL conditions.
But for real deployment, you still need to measure four things:
- your model
- your inference runtime
- your context length and concurrency
- your background load and thermal state
The question I care about is not whether Thor can reproduce the highest official number in a clean benchmark. It is whether Thor can keep useful latency when cameras, RAG, agents, and local services are all running on the same edge system.
That is the benchmark edge AI actually needs.