MultimodalFlow
中文
← Back to Blog

Edge AI Deployment Checklist for Small Teams in 2026

edge AIdeploymentJetsonproductionchecklist

Deploying AI at the edge is meaningfully different from deploying cloud services. The hardware is constrained. The network may be unreliable or absent. You can't just spin up another instance when things go wrong. And the system often runs unattended for months.

This checklist comes from deploying vision AI systems on Jetson devices, industrial PCs, and edge servers over the past few years. It covers the things that bite you in production but rarely appear in tutorials.

1. Hardware Selection

Nail down VRAM / unified memory requirements before buying.

For Jetson devices (Orin NX, AGX Orin, AGX Thor), the "unified memory" figure is shared between CPU and GPU. If you plan to run a 7B language model alongside a vision pipeline, budget accordingly. A Jetson Orin NX 16GB leaves you very little headroom once the OS, vision pipeline, and LLM are all loaded.

For GPU servers, the rule of thumb: your model's FP16 footprint should be ≤ 60% of VRAM. Leave room for the KV cache if you're running inference on long sequences, and for the OS display driver if you're running a desktop environment.

Match compute capability to latency requirements.

Real-time inference (≤ 30ms per frame) at 1080p with a 30B-parameter model is simply not achievable on Jetson Orin NX. Know what your model requires before committing to hardware. Run trtexec benchmarks or Ollama speed tests before the hardware ships to site.

Check software compatibility.

The CUDA version on your target device constrains which versions of PyTorch, TensorRT, and ONNX Runtime you can use. JetPack versions and CUDA versions are tightly coupled on Jetson. Check the compatibility matrix before finalizing your stack.

2. Model Optimization

Always export to ONNX and validate the export.

Compare outputs between your PyTorch model and the ONNX export on a set of known inputs before building the TensorRT engine. Numerical differences ≥ 1e-3 indicate an export problem, usually caused by unsupported ops or custom modules.

Use the right quantization for your accuracy budget.

  • FP16: Almost always safe. Use this by default.
  • INT8: Requires calibration. Budget 4–8 hours to collect calibration data and validate accuracy on your test set.
  • Dynamic quantization (GGUF Q4_K_M, etc.): For LLMs on constrained hardware, Q4 is typically the sweet spot between speed and quality.

Build TensorRT engines on the target device.

The engine is GPU-architecture specific. A TRT engine built on RTX 3090 will not load on Jetson AGX Orin. Either build on the target or use a device with the same CUDA compute capability.

Set explicit input shapes.

Dynamic shapes add overhead at inference time. If your input resolution is fixed (e.g., 1280×720), specify it at engine build time. You get a faster, smaller engine.

3. Thermal Management

This section causes the most production surprises.

Jetson will throttle under sustained load.

All Jetson devices have thermal protection. If the SoC temperature exceeds the threshold, clock speeds drop automatically. A model that benchmarks at 60 FPS in a lab may run at 35 FPS after 20 minutes on a production line without adequate cooling.

What to do:

  • Run a 30-minute sustained inference test while monitoring temperature: tegrastats | grep -o 'CPU@[0-9.]*C'
  • Install an active cooling solution (fan heatsink) for any deployment where sustained throughput matters
  • Set the Jetson power mode explicitly: nvpmodel -m 0 (max performance) or choose the appropriate mode for your thermal envelope

Avoid thermal cycling on RTX GPUs.

On an RTX 3090 running inference in a server rack, ensure adequate airflow. GPU junction temperatures above 90°C will trigger throttling. Add watch -n1 nvidia-smi to your deployment runbook to check during commissioning.

4. System Reliability

Design for restart.

Your inference process will eventually crash — model loading error, CUDA OOM, corrupted frame from camera SDK. Use systemd or Docker restart policies so the service recovers automatically without human intervention.

# /etc/systemd/system/inference.service
[Service]
ExecStart=/usr/bin/python3 /opt/inference/run.py
Restart=always
RestartSec=5

Handle camera disconnects.

USB cameras disconnect. GigE cameras lose packets. Your inference loop must handle None frames and camera reconnection gracefully. The naive implementation that crashes on camera.read() returning False will page you at 3 AM.

Log structured telemetry, not just print statements.

In production you need to answer: "What was the inference latency at 14:32 yesterday?" Log timestamps, inference time, confidence scores, and any error conditions to a file (or push to a metrics service). Plain print() calls are ungreppable.

5. Model Versioning and Updates

Version your TensorRT engines alongside your model weights.

An engine built from model v1.2 will silently produce wrong outputs if you swap in model v1.3 weights without rebuilding. Keep engines and weights in sync. Use a naming convention like model_v1.3_orin_fp16.engine.

Test on real production data before rolling out.

Distribution shift is real. A model that achieves 94% accuracy on your test set may drop to 78% on production line images under fluorescent lighting with a smudged lens. Collect 200–500 production samples and evaluate before each major model update.

Keep a rollback path.

Store the previous working engine and inference code. When a new model update fails on production hardware (it will), you need to revert in minutes, not hours.

6. Pre-Deployment Verification Checklist

Before sending hardware to site:

  • Inference process starts on boot (systemd/Docker restart confirmed)
  • 30-minute sustained inference test completed without throttling
  • Camera disconnect / reconnect handled (unplug and replug during test)
  • Model accuracy validated on production-representative data
  • Telemetry logging writing to disk and readable
  • Remote access confirmed (SSH, VPN, or remote desktop)
  • Storage: at least 20 GB free after all models are installed
  • NTP time sync enabled (logs need accurate timestamps)
  • Watchdog or health check endpoint running

My Take

The items on this list are not edge cases — they're the things that actually cause production incidents. The 30-minute thermal test sounds tedious, but I've shipped two systems that ran fine in the office and throttled badly at the customer site because the enclosure had no ventilation.

The restart policy feels like overkill until your inference process crashes at midnight because the camera sent a malformed frame.

Edge AI deployment is engineering, not just ML. The model is the easy part.

This checklist reflects deployment experience with Jetson Orin NX, Jetson AGX Thor, and RTX 3090-based edge servers. Some items are Jetson-specific; the principles apply broadly.