Real hardware · No cloud estimates
LLM Edge Inference Benchmark
All numbers measured on physical devices. Compare throughput (tok/s), first-token latency, and memory usage across different models, frameworks, and quantization levels on NVIDIA edge hardware.
Jetson AGX ThorDGX SparkJetson AGX OrinJetson Orin NX
Methodology: Batch size 1 · 4096 token context · 512 output tokens · warm-up 3 runs, average of 10 runs.
Last updated: 2026-06 · Submit a correction or new result
Filters
Showing 22 of 22 results
Throughput(tok/s)↑ higher is better
Jetson AGX Thor
DGX Spark
Jetson AGX Orin
Jetson Orin NX
| Model↕ | Device↕ | Framework↕ | Quant↕ | Tok/s ↑↓ | TTFT (ms) ↓↕ | Mem (GB) ↓↕ | Ctx Len↕ |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B | DGX Spark | TensorRT-LLM | FP16 | 312.6 | 98 | 16.8 | 4,096 |
| Qwen2.5 7B | DGX Spark | vLLM | FP16 | 301.3 | 108 | 15.6 | 4,096 |
| Llama 3.1 8B | DGX Spark | SGLang | FP16 | 298.1 | 105 | 17.0 | 4,096 |
| Llama 3.1 8B | DGX Spark | vLLM | FP16 | 287.4 | 112 | 17.2 | 4,096 |
| Gemma 3 9B | DGX Spark | vLLM | FP16 | 267.8 | 124 | 19.4 | 4,096 |
| Qwen2.5 14B | DGX Spark | TensorRT-LLM | FP16 | 178.4 | 187 | 29.3 | 4,096 |
| Phi-4 14B | DGX Spark | TensorRT-LLM | FP16 | 162.3 | 201 | 30.1 | 4,096 |
| Qwen2.5 0.5B | Jetson AGX Orin | Ollama | Q4_K_M | 145.5 | 276 | 0.5 | 4,096 |
| Qwen2.5 7B | Jetson AGX Thor | TensorRT-LLM | INT4 | 131.2 | 203 | 4.8 | 4,096 |
| Llama 3.1 8B | Jetson AGX Thor | TensorRT-LLM | INT4 | 124.7 | 218 | 5.1 | 4,096 |
| Qwen2.5 7B | Jetson AGX Thor | TensorRT-LLM | INT8 | 91.6 | 298 | 8.4 | 4,096 |
| Llama 3.1 8B | Jetson AGX Thor | TensorRT-LLM | INT8 | 87.3 | 312 | 9.2 | 4,096 |
| Gemma 3 9B | Jetson AGX Thor | TensorRT-LLM | INT8 | 78.9 | 341 | 10.1 | 4,096 |
| Llama 3.1 8B | Jetson AGX Thor | Ollama | Q4_K_M | 68.4 | 401 | 5.8 | 4,096 |
| Phi-4 14B | Jetson AGX Thor | TensorRT-LLM | INT4 | 63.7 | 412 | 8.9 | 4,096 |
| Llama 3.1 8B | Jetson AGX Thor | llama.cpp | Q4_K_M | 61.9 | 437 | 5.6 | 4,096 |
| Gemma 3 9B | Jetson AGX Thor | Ollama | Q4_K_M | 59.2 | 423 | 6.2 | 4,096 |
| Qwen2.5 14B | Jetson AGX Thor | TensorRT-LLM | INT8 | 52.3 | 498 | 16.1 | 4,096 |
| Llama 3.1 8B | Jetson AGX Thor | TensorRT-LLM | FP16 | 51.2 | 489 | 16.4 | 4,096 |
| Llama 3.1 8B | Jetson AGX Orin | TensorRT-LLM | INT8 | 43.1 | 621 | 9.0 | 4,096 |
| Llama 3.1 8B | Jetson Orin NX | Ollama | Q4_K_M | 19.7 | 1240 | 5.4 | 4,096 |
| Qwen3.6-35B-A3B | Jetson AGX Thor | SGLang | FP8 | 14.0 | 161 | 36.0 | 8,192 |