Jetson AGX Thor + Orbbec 3D Camera: Building a Local Multimodal Robot Inspection System

Development period: June 1, 2026 to July 2, 2026.
Hardware: NVIDIA Jetson AGX Thor Developer Kit with an Orbbec Gemini 345Lg / RealSense D435i 3D camera, plus a reserved data channel for an Airy LiDAR.
Project code location: NVIDIA_Thor_Demos/thor_orbbec_3d_robot_demo.

This article is a retrospective of an edge 3D robot perception project running on Jetson AGX Thor: one Thor plus one 3D camera, completing RGB detection, depth/point cloud processing, occupancy grid building, risk assessment, VLM scene understanding, RAG rule retrieval, LLM decisioning, TTS announcements, and inspection report generation — all locally.

My starting point for this project was simple: many robotics solutions look impressive in demo videos, but once they leave cloud APIs, desktop GPUs, and pre-recorded footage, they struggle to run reliably on site. Real showrooms, factories, warehouses, and labs need a different kind of capability: the device itself can see, understand, decide, and record — and keeps working when the network is unreliable.

1. Project Overview and Target Scenarios

The goal of the first version of this project can be summed up in one sentence:

Let a robot complete the full loop of "see the environment, understand the risk, look up the rules, make a decision, speak the result, and leave a report" locally on Jetson AGX Thor.

The image below shows RGB-D and point cloud data captured by the real device service on Jetson AGX Thor. The RGB frame contains an office environment, so it was blurred for privacy before publishing; the depth map and the top-down point cloud view keep the spatial structure, demonstrating that the system really ingests 3D sensor data rather than just simulating things in the frontend.

The project targets mobile robots, inspection robots, and factory aisle safety monitoring. A typical task: when a person, a box, a tool cart, or clutter appears in the robot's path, the system must decide whether it is safe to proceed. With RGB images alone, the robot may know "there is a person in the frame" but not the distance. With depth or point clouds alone, the robot knows "there is an obstacle 1.05 meters ahead" but not what the obstacle is. With rule scripts alone, it is hard to explain the on-site semantics.

So I split the system into three tracks:

1. Geometric perception: the RGB-D camera outputs RGB, Depth, IR, and point clouds; the point cloud module computes the nearest obstacle distance, ground/non-ground point counts, and aisle occupancy.
2. Semantic understanding: YOLO handles object detection, and a VLM interprets scene semantics, e.g. "a person is about 2.4 m ahead" or "a box occupies the right side of the aisle".
3. Decision and explanation: RAG retrieves constraints from local SOP / inspection rules, and the LLM generates robot action recommendations based on detection results, point cloud distances, the VLM description, and the rules.

The final output is not a vague "there is a risk" but a structured inspection report: what targets were detected, how far the nearest obstacle is, what the risk level is, why the robot should stop, and what the next action is.

The first version supports fallback modes for sensors and models, so the full business workflow can run offline even without a camera, real large models, or a complete sensor chain. But the code already reserves switch points for RealSense, the Orbbec SDK, YOLO/TensorRT, Qwen2.5-VL, NVlabs Eagle LocateAnything, Ollama/vLLM, and TTS backends. In other words, it is not a pure demo page — it is an edge robot perception skeleton whose modules can be progressively replaced with real ones.

2. Hardware, SDK, Model, and Tooling Choices

1. Hardware platform: why Jetson AGX Thor

I chose Jetson AGX Thor mainly because this project consumes three kinds of resources at once:

• 3D sensor data: RGB, Depth, IR, and point clouds refresh continuously, stressing I/O, CPU, and memory.
• Multi-model inference: YOLO, VLM, LLM, and TTS can run serially, with parallel optimization added over time.
• Local services: FastAPI, WebSocket, frontend visualization, report generation, and monitoring all run on the same device.

Thor's strength is not running a single model in isolation, but hosting the entire multimodal robot pipeline on one edge device. Compared to sending frames to the cloud, processing locally on Thor reduces network dependence and keeps factory, lab, or warehouse footage inside the intranet. For robots, low latency and a closed data loop matter more than raw model parameter counts.

The project also reuses Jetson Orin experience. For example, the monitoring module supports nvidia-smi, pynvml, and the tegrastats approach common on Jetson, making it easier to later port a trimmed-down version of the pipeline to AGX Orin or Orin NX.

2. Sensors and system software

The sensor side has three layers of compatibility:

Module	Role	Current status
RealSense D435i	RGB-D camera input	Service wrapper implemented, tried first at startup
Orbbec Gemini 345Lg	Target 3D camera	OrbbecCameraService / SDK adapter interface reserved
MockOrbbecCameraService	Offline debugging and workflow validation	Default fallback, keeps the system runnable without hardware
Airy LiDAR	LiDAR point cloud extension	Point cloud and sector analysis interfaces reserved

The software stack is centered on Python + FastAPI:

• FastAPI: Web UI, REST API, image upload, inspection tasks, health checks, and a WebSocket event stream.
• Pydantic: structured schemas for detection results, point cloud statistics, risk analysis, VLM output, and LLM decisions.
• NumPy / OpenCV: depth map, IR, point cloud visualization, and safety-zone heatmap processing.
• psutil / pynvml / nvidia-smi: CPU, memory, GPU, power, and temperature metrics.
• Markdown reports: an archivable .md report is generated automatically after every inspection run.

3. AI model choices

The current version uses fallback AI services by default. The contest article emphasizes the full development process and edge system architecture, so in the first phase I prioritized a runnable pipeline, stable data structures, and clean module boundaries. The real model integration points are:

Module	Default implementation	Recommended real backends
Object detection	Replaceable detection service	Ultralytics YOLO, TensorRT Engine, DeepStream
VLM	Switchable vision-language service	Qwen2.5-VL, InternVL, NVlabs Eagle LocateAnything
RAG	MarkdownRAGService	FAISS, Chroma, local SOP document library
LLM decisioning	Replaceable decision service	Ollama, vLLM, OpenAI-compatible API
TTS	Replaceable speech service	Piper, edge-tts, CosyVoice

I deliberately did not wire every model to real inference from day one, because the biggest risk in robot systems is "the model runs, but the engineering pipeline is unstable". Locking the input/output schemas first and then swapping in real backends prevents the frontend, state machine, reports, and decision layer from being repeatedly dragged around by model output formats.

3. Full Development Workflow and Core Implementation

1. Task state machine

The main flow is driven by InspectionTaskRunner. Every inspection task goes through these stages:

CAPTURE_SENSOR_FRAME
RGB_DETECTION
DEPTH_PROCESSING
POINTCLOUD_PROCESSING
OCCUPANCY_GRID_BUILDING
RISK_ANALYSIS
VLM_ANALYSIS
RAG_RETRIEVAL
LLM_DECISION
TTS_SPEAKING
REPORTING
DONE

The benefit is that every stage has explicit inputs, outputs, and timing records. The frontend subscribes to task events over WebSocket and can watch the task progress from camera capture through YOLO detection and point cloud processing all the way to report generation. During development, whichever stage is slow, throws errors, or produces abnormal output can be located immediately.

2. Camera and input source switching

Robot perception projects share a common problem: you don't always have the real sensor at hand during development. If the system hard-depends on a live camera, much of the functionality can't be debugged. This project's input source supports two modes:

• Camera mode: uses the current frame from the RealSense / Orbbec / fallback camera service, keeping the RGB, Depth, IR, and PointCloud flow.
• Image mode: uploads a local image as the RGB / VLM input, while Depth, IR, and PointCloud are still filled in by the current sensor or fallback data.

This makes development, testing, and on-site validation easier. With a camera on site, run the real-time pipeline; without one, upload a single aisle photo and still fully validate detection, risk analysis, RAG, LLM, and reporting.

3. Point cloud statistics and occupancy grid

The point cloud processing does not attempt full SLAM. It first solves the most immediate problems in robot obstacle avoidance: is there an obstacle ahead, how far is the nearest one, and is the aisle occupied.

The system range-filters and downsamples the point cloud, capping it at 8,000 points by default — enough to preserve spatial structure without overloading browser-side rendering. It then computes:

• Total point count and downsampled count
• Nearest obstacle distance
• Ground point count
• Non-ground point count
• Passable and occupied area statistics

Results from one test report:

Metric	Value
Point cloud size	8,500
Downsampled size	8,000
Nearest obstacle distance	1.05 m
Obstacle points	2,948
Ground points	5,552
Occupancy grid occupied	35
Occupancy grid free	803

This data is already enough for a real robot to conclude: there is an obstacle within 2 meters ahead, and the aisle should not be traversed directly.

To make the data easier to share publicly, I rendered the point cloud returned by the API as a top-down distance map. The robot sits at the bottom of the image; red points mark close-range high-attention areas, and teal points show spatial structure farther away. Compared to showing only the RGB frame, the top-down point cloud view better illustrates what the 3D perception module contributes to obstacle avoidance.

4. Risk analysis, RAG, and LLM decisioning

The risk analysis module first produces a preliminary judgment from hard rules. For example:

• An obstacle within 2 meters ahead → slow down or stop.
• A person entering the 3-meter safety zone ahead → stop and wait.
• Boxes, tools, or clutter occupying the aisle → mark the aisle as blocked.
• When RGB and point cloud conflict, trust the point cloud distance.

These rules live in local Markdown SOPs, are retrieved by the RAG module, and are handed to the LLM. The LLM does not decide "by gut feeling" — it receives structured context:

• User instruction: locate the person and the box, and judge whether the robot can pass.
• YOLO detections: person at confidence 0.91, about 2.4 m away; box at confidence 0.88, about 1.3 m away.
• Point cloud statistics: nearest obstacle about 1.05 m.
• VLM description: a person and a box are in the aisle ahead; the box is on the right and may block passage.
• RAG rules: stop when a person enters the 3-meter zone; slow down or stop for obstacles within 2 meters.

The final decision output:

{
  "decision": "STOP_AND_WAIT",
  "speed_advice": "Stop or wait at low speed",
  "path_advice": "Do not proceed directly; wait for the person to leave, then detour on the left",
  "reasoning_summary": "Point cloud shows the nearest obstacle at about 1.1 m; RGB detected a person at about 2.4 m. Per the rules, the robot must stop and wait when a person enters the 3-meter zone ahead.",
  "final_message": "A person and an obstacle are ahead. Recommend stopping, waiting, and replanning the path."
}

This result matters for robot systems. It carries both an action recommendation and the supporting rationale, which helps engineers review incidents and eases later integration with ROS2, Isaac Sim, or a real chassis controller.

4. Development Challenges, Debugging, and Solutions

Challenge 1: keeping the project runnable when real hardware is flaky

If any single module — 3D camera, depth stream, IR stream, point cloud, or model service — fails, the whole pipeline can break. My approach is "real services first, fallback services guarantee the flow". At startup the system tries RealSense first, then Orbbec, and automatically falls back to the local mock camera service on failure.

This is not laziness — it decouples the engineering debugging: the frontend, state machine, reports, RAG, and decisioning can each be validated independently. Once the camera SDK is installed, only the camera service is swapped; the upper layers stay untouched.

Challenge 2: multimodal output formats spiral out of control

YOLO, VLM, and LLM outputs are inherently unstable. If raw model text is passed straight to the frontend, the pages and reports quickly become a pile of special cases. All core outputs in this project are constrained by Pydantic schemas — Detection, PointCloudStats, RiskAnalysis, VLMSceneResult, RobotDecision, and so on.

When real Qwen2.5-VL or vLLM backends are wired in later, their outputs only need to be validated against the same schemas. On validation failure the system retries or degrades, instead of letting errors propagate to the frontend.

Challenge 3: point clouds must be both useful and lightweight

Point cloud data grows quickly. Returning the full point cloud to the browser through the API slows the page down and adds CPU pressure on the Jetson. I used three steps:

1. Filter by distance, keeping only the business-relevant range.
2. Downsample to at most 8,000 points.
3. Keep frontend rendering lightweight; the real judgments happen on the backend.

This preserves the 3D spatial feel without letting point cloud visualization crush the main pipeline.

Challenge 4: robot decisions must be explainable

Many AI applications end with a single sentence: "recommend stopping". In robotics, that is not enough. Engineers and customers will ask: why stop? Because of the person, or the box? At what distance? Where did the rule come from?

So the project writes the RAG rules, risk analysis, VLM scene description, LLM reasoning summary, and the final action into the report. After each inspection run, the report shows the full evidence chain, not just the final answer.

5. What This Project Validates Through the Lens of NVIDIA Vesta

During this project, I also paid close attention to NVIDIA Vesta and the broader direction of embodied reasoning / generalist planners. Vesta's core idea is not to let one model directly replace the low-level controller. Instead, it unifies localization, navigation, embodied reasoning, long-horizon planning, and memory into a high-level robot planner. The planner observes the environment, combines it with task goals and historical memory, then outputs the next natural-language subtask for a low-level actor or control system to execute.

This article does not claim to reproduce Vesta's paper results. The reason is straightforward: this project does not run Vesta weights, nor does it reproduce the R2R-CE, AgiBot, or real bimanual robot evaluations. A more accurate framing is that this Jetson Thor + RGB-D robot inspection project validates one engineering direction represented by Vesta: converging spatial perception, rule memory, scene understanding, and action planning into a local high-level planner interface.

From that perspective, the project validates three things.

First, it validates the engineering feasibility of a unified planner interface. YOLO detections, point cloud distances, occupancy grids, VLM descriptions, RAG rules, and LLM decisions are not shown as isolated demos. They are organized into one structured context and converted into robot action recommendations such as STOP_AND_WAIT, SLOW_DOWN, and REROUTE. This is close to Vesta's high-level planner role: the planner does not directly drive motors; it proposes the next executable subtask or action intent.

Second, it validates the value of 3D spatial grounding for robot inspection. A pure RGB VLM can say "there is a person and a box in the image", but RGB-D and point clouds can add spatial evidence: the nearest obstacle is about 1.05 m away, the person is about 2.4 m away, and the box is about 1.3 m away. For robots, whether a path is traversable is not a purely semantic question. It depends on both semantics and geometry.

Third, it validates the importance of local rule memory and explainable decisions. Vesta emphasizes historical observations and previously executed subtasks for long-horizon planning. This project currently uses RAG and structured reports to preserve rules, risk evidence, and decision results. A natural next step toward a more Vesta-like setup is continuous inspection memory: for example, the first pass records a box blocking the right side of the aisle; in the second pass the box is visually occluded, but the point cloud still shows a right-side obstacle; the planner can then combine historical action context with the current 3D observation and still recommend stopping or rerouting.

So this project is better described as an edge-side Vesta-style architecture validation, not a Vesta model reproduction. It shows that on an edge device like Jetson Thor, 3D perception, vision-language understanding, rule retrieval, and action decisioning can be chained into a stable local loop. When Vesta or similar embodied planners become available as deployable weights, the current LLM decision layer can be replaced by a stronger unified planner.

6. Results, Data, and Next Steps

With the fallback sensor and default service configuration, one complete inspection run produces the following locally:

Module	Output
RGB detection	person and box
Nearest obstacle (point cloud)	1.05 m
Risk level	HIGH
Stop required	true
VLM scene conclusion	A person and a box in the aisle may block passage
RAG matched rules	Slow/stop for obstacles within 2 m; stop and wait for people within 3 m
LLM decision	STOP_AND_WAIT
Report output	Markdown inspection report generated automatically

Stage timings are also recorded in the report. In one test, sensor frame capture took about 4.26 ms and the RGB detection stage about 201.24 ms. Since the current YOLO stage uses the fallback detection service, these timings mostly reflect pipeline scheduling and the end-to-end flow; after TensorRT YOLO is integrated, the next step is to benchmark real detection FPS, end-to-end latency, and power consumption.

The project has several clear optimization directions:

1. Integrate the real Orbbec SDK: adapt real-time RGB, Depth, IR, and PointCloud streams for the Gemini 345Lg.
2. Integrate TensorRT YOLO: replace the fallback detector with a real model and record FP16 / INT8 inference speeds.
3. Integrate a real VLM: test Qwen2.5-VL, InternVL, and LocateAnything-3B visual grounding on Thor.
4. Integrate a local LLM: run Qwen-series models with Ollama or vLLM and validate structured JSON output stability.
5. Integrate ROS2 / Isaac: map actions like STOP_AND_WAIT, SLOW_DOWN, and REROUTE to robot control interfaces.
6. Profile power and thermals: record GPU utilization, memory usage, power draw, and temperature over long runs.

Conclusion

My biggest takeaway from building this Jetson AGX Thor + Orbbec 3D robot inspection system: the key to edge AI projects is not just "can it run a model", but whether sensors, models, rules, decisions, interaction, and reports can be chained together reliably.

That is where Thor's value lies. It lets a robot system avoid sending all data to the cloud, avoid depending on remote large-model services, and avoid stalling when the network fluctuates. For factories, warehouses, labs, and showrooms, what really works is the local closed loop: process data on site, judge risk on site, explain results on site.

This project is still iterating. The first version establishes the engineering skeleton and the closed loop; real cameras, real detection models, real VLMs, and real LLMs will be swapped in step by step. Once all of those are connected, it will settle into an edge multimodal robotics template that can be reused across mobile robots, inspection robots, and industrial safety monitoring terminals.