Blog
Notes on edge AI engineering and practical deployment.
Qwen3.8-27B Benchmarked: 58 tok/s on an RTX 3090 at Q4, 4.1 tok/s on a Jetson Thor at BF16
The same 27B model, two machines, a 14× speed gap. Real numbers from an RTX 3090 (Ollama Q4_K_M) and a Jetson AGX Thor (vLLM BF16) — how much of the gap is quantization, how much is memory bandwidth, and what Thor wins back under concurrency.
Qwen3.8benchmarkRTX 3090Jetson ThorvLLMOllamaedge inferencememory bandwidth2×Jetson Thor for DeepSeek-V4-Flash-NVFP4: QSFP28 Preflight and Real Blockers
A preflight test for running nvidia/DeepSeek-V4-Flash-NVFP4 across two Jetson Thor systems. The result: SGLang/vLLM containers are present, but QSFP28 IP configuration, disk capacity, and Jetson aarch64 NVFP4 maturity must be solved before downloading and launching the model.
DeepSeek-V4-FlashNVFP4Jetson ThorQSFP28SGLangvLLMDistributed InferenceEdge AICan DeepSeek's New V4-Flash-0731 Run on Jetson Thor? The 284B MoE Edge Boundary
DeepSeek-V4-Flash-0731 is a recent MIT-licensed open-weight model with 284B total parameters, 13B activated parameters, and 1M context. This post checks whether it fits Jetson Thor and outlines a practical benchmark path.
DeepSeek-V4DeepSeek-V4-FlashJetson ThorEdge AIMoESGLangvLLMLocal InferenceDeepSeek V4 Flash NVFP4 on 2×Jetson Thor, Debug Log 1: SGLang Reaches DeepGEMM
A practical debug log for trying nvidia/DeepSeek-V4-Flash-NVFP4 across two Jetson Thor systems: QSFP28 setup, model download, SGLang nightly, KV cache fallback, Docker runtime fixes, and the final DeepGEMM HyperConnection sm110 blocker.
DeepSeek-V4DeepSeek-V4-FlashNVFP4Jetson ThorSGLangvLLMQSFP28Edge AICan Qwen3.8-Max Run on Jetson Thor? The 2.4T Frontier Model vs Edge Hardware
Qwen3.8-Max is Alibaba's latest Qwen flagship, described as a 2.4T-parameter model for coding and complex work. This post checks the Jetson Thor boundary and explains why Qwen3-8B or Qwen3.6 models are the realistic local benchmark targets.
Qwen3.8-MaxQwen3-8BJetson ThorEdge AILLMSGLangllama.cppLocal InferenceJetson T2000 vs T3000 vs Orin: 2026 Edge AI Hardware Guide
How should you choose between NVIDIA Jetson T2000, T3000, AGX Thor, Orin, and desktop GPUs for edge AI in 2026? A practical guide for LLMs, VLMs, robotics, and multimodal deployment.
JetsonT2000T3000OrinThoredge AIroboticsVLMTensorRT Edge-LLMhardware guide2026MiniMax H3 on Jetson AGX Thor: Early Public Benchmark Plan for FL2VA, Ref2VA, and INT8 ConvRot
A reproducible MiniMax H3 benchmark protocol on Thor-JD covering FL2VA, Ref2VA, T2VA, INT8 ConvRot, generation time, output validity, memory capacity, and scripts.
MiniMax H3Hailuo 3.0Jetson ThorFL2VARef2VAINT8 ConvRotSGLangComfyUIbenchmarkedge AIAre NVIDIA Jetson AGX Thor LLM Benchmarks Realistic? I Retested on a Live Deployment
NVIDIA has published Jetson AGX Thor vLLM benchmark numbers for Llama, Qwen, DeepSeek, and Qwen2.5-VL. I retested Qwen2.5-7B on a live Thor test machine running model serving, retrieval services, and a desktop environment, then explain why official benchmarks differ from deployment results.
JetsonThorLLMVLMvLLMbenchmarkedge inferenceNVIDIAdeploymentJetson AGX Thor + Orbbec 3D Camera: Building a Local Multimodal Robot Inspection System
A hands-on retrospective of a 3D robot inspection project built on NVIDIA Jetson AGX Thor with Orbbec Gemini 345Lg / RealSense D435i, point clouds, YOLO, VLM, RAG, and a local LLM.
JetsonJetson ThorRobotics3D VisionOrbbecVLMRAGEdge AIJetson Device Skills: Agentic Edge AI on AGX Orin 64GB — Install & Live Test
NVIDIA's open-source Jetson Device Skills give Claude Code and other agents native Jetson workflows. Hands-on install and benchmark on a real AGX Orin 64GB.
JetsonOrinedge AIClaude CodeNemoClawLLMbenchmarkJetPackagentdeploymentQwable-v1 on NVIDIA Thor: Full Deployment, Testing & Evaluation
End-to-end guide for deploying Qwable-v1 (Qwen3.6-35B MoE + Claude Fable-5 distill) on NVIDIA Jetson AGX Thor — download, SGLang serving, Web UI, and closed-loop Agent benchmark. All numbers from live hardware.
JetsonThorQwableLLMSGLangAgentFable-5distillationedge inferencedeploymentClustering Two NVIDIA DGX Spark Systems: 200GbE Looks Fast, but NCCL Tells the Real Story
Two DGX Spark nodes connected over 200GbE / ConnectX-7 RDMA. Raw ib_write_bw hits ~197 Gb/s, but real NCCL collective bandwidth lands at 10 GB/s — and here's exactly why they differ.
DGX SparkNCCLRDMAvLLMmulti-node200GbERoCEdistributed inferenceLLMConnectX-7Qwen3.6-27B on RTX 3090 vs Jetson Thor: June 2026's Best Dense Coding Model, Benchmarked
Qwen3.6-27B outperforms 397B MoE models on coding benchmarks and fits on a single RTX 3090. Real inference numbers on RTX 3090 (Ollama) and Jetson Thor (llama.cpp) — and why June 2026's hottest models like Kimi K2.7-Code need very different hardware.
Qwen3.6benchmarkRTX 3090Jetson ThorLLMOllamallama.cppcodingedge inferenceEdge AI Deployment Checklist for Small Teams in 2026
A practical checklist for deploying AI inference systems on edge hardware — covering hardware selection, model optimization, thermal management, monitoring, and the failure modes that only appear in production.
edge AIdeploymentJetsonproductionchecklistTensorRT for Computer Vision: What Actually Speeds Up Inference on Jetson and Desktop GPUs
A practical guide to TensorRT optimization for vision models — what it does, where the real speedups come from, which layers benefit most, and how to avoid the common pitfalls when deploying on Jetson or RTX GPUs.
TensorRTJetsoninference optimizationcomputer visiondeploymentQwen3 30B vs Gemma 4 26B on RTX 3090: Full VRAM Showdown
Head-to-head benchmark of Qwen3 30B and Gemma 4 26B on a single RTX 3090 with full VRAM available. Real Ollama API measurements — generation speed, prefill speed, and which model wins at each task.
Qwen3Gemma4benchmarkRTX 3090LLMOllamalocal inferenceGemma 4 on RTX 3090: 4B vs 12B vs 26B Benchmark — What Happens When VRAM Runs Out
Live benchmark of all three Gemma 4 sizes (4B, 12B, 26B) on a single RTX 3090. Shows how VRAM overflow kills generation speed on the 26B model, and which size actually makes sense for 24GB cards.
Gemma4benchmarkRTX 3090LLMVRAMOllamalocal inferenceGemma 4 12B Benchmark: Jetson AGX Thor vs RTX 3090
Real hardware benchmark of Google's Gemma 4 12B multimodal model on two platforms: NVIDIA Jetson AGX Thor (edge AI) and RTX 3090 (desktop GPU). Tokens per second, prefill speed, and memory from live hardware.
Gemma4benchmarkJetsonThorRTX 3090LLMedge inferencellama.cppLocateAnything-3B Benchmark: RTX 3090 vs Jetson AGX Thor
Real hardware benchmark of NVIDIA's LocateAnything-3B vision-language model across RTX 3090 and Jetson AGX Thor — inference speed, memory, and deployment conclusions for edge AI.
locateanythingnvidiajetson thorrtx 3090benchmarkedge aivision language modelobject detectionJetson AGX Thor LLM Benchmark (2026): Qwen3.6-35B and Qwen2.5-1.5B Real-Hardware Results
Real inference benchmarks on NVIDIA Jetson AGX Thor — Qwen3.6-35B-A3B FP8 via SGLang and Qwen2.5-1.5B Q4_K_M via llama.cpp. Tokens per second, TTFT, and memory usage from live hardware.
JetsonLLMbenchmarkedge inferenceQwenSGLangllama.cppThorJetson Thor vs Jetson Orin 64GB: Local LLM Benchmark Notes
Hands-on local LLM benchmark notes from Jetson Thor and Jetson Orin 64GB, including Qwen model setup, latency, tokens per second, and practical deployment advice.
JetsonThorOrinLLMbenchmarkQwenSGLangOllamaLLM Inference Benchmarks on Jetson AGX Orin 64GB (2026)
Real hardware benchmarks for Llama 3.1 8B, Qwen2.5 7B, and Phi-3 Mini on a Jetson AGX Orin 64GB — tokens per second, TTFT, memory usage, and practical deployment notes.
JetsonLLMedge inferencebenchmarkOrinllama.cpp