TensorRT for Computer Vision: What Actually Speeds Up Inference on Jetson and Desktop GPUs

TensorRT gets cited constantly in edge AI deployment discussions, but the descriptions are usually vague: "it's faster," "it's NVIDIA's inference engine," "it compresses your model." None of that tells you what to actually expect or where the gains come from.

This post covers TensorRT from a deployment engineer's perspective — what it does mechanically, which model architectures benefit most, how quantization works in practice, and the parts that can go wrong.

What TensorRT Actually Does

TensorRT is an inference optimizer and runtime for NVIDIA GPUs. You give it a trained model (from PyTorch via ONNX, or TensorFlow), and it produces an optimized engine file tuned specifically for your GPU hardware and target precision.

The core operations it performs:

Layer fusion. Operations that are mathematically adjacent — like a convolution followed by batch normalization followed by ReLU — get merged into a single kernel call. This eliminates the round-trips to GPU memory between operations. On a typical ResNet block, layer fusion alone can cut the number of kernel launches by 60–70%.

Kernel auto-tuning. TensorRT benchmarks multiple CUDA kernel implementations for each operation against your specific GPU and picks the fastest one. A convolution that runs fast on an A100 might not be the fastest on a Jetson AGX Xavier. The engine is hardware-specific.

Precision reduction. FP32 → FP16 cuts memory bandwidth by half on supported hardware (all modern NVIDIA GPUs). INT8 cuts it by 75% but requires calibration data to minimize accuracy loss.

Memory planning. TensorRT analyzes the entire graph to find which tensors can share memory buffers because their lifetimes don't overlap. This reduces peak VRAM usage, which matters significantly on constrained edge devices.

Where the Speedups Come From

The exact speedup depends heavily on the model architecture. Here's what I've observed across several projects:

Convolution-heavy models (YOLO, ResNet, EfficientDet): Biggest gains. Layer fusion and INT8 quantization can yield 3–5× throughput improvement over native PyTorch on the same GPU. On Jetson AGX Orin, a YOLOv8m exported to TensorRT INT8 typically runs at 120–180 FPS vs 35–50 FPS in PyTorch.

Transformer-based vision models (ViT, DETR): More modest gains, typically 1.5–2.5×. The attention mechanism is harder to fuse efficiently, and the speedup depends on sequence length.

Detection heads with dynamic output shapes: These require special handling. TensorRT needs fixed input/output shapes at engine build time unless you use dynamic shape profiles, which adds complexity.

The Quantization Reality

FP16 (half precision) is usually a free lunch. Every NVIDIA GPU from Maxwell onwards has hardware support for FP16 tensor operations. The accuracy impact is minimal for vision models — typically less than 0.5% mAP drop on detection tasks.

INT8 is more complex. You need a calibration dataset — usually 500–1000 representative images — that TensorRT runs through the network to determine the optimal scaling factors for each layer. If your calibration set doesn't represent your actual inference distribution, INT8 accuracy will degrade noticeably.

The common failure mode: engineers calibrate on the training dataset but deploy on images from a different camera, lighting condition, or geographic location. Always calibrate on data that matches your deployment scenario.

Some layers don't benefit from INT8 and TensorRT will automatically fall back to FP16 for them. You can inspect this in the TensorRT build log.

Building a TensorRT Engine: The Actual Steps

import tensorrt as trt
import numpy as np

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)

def build_engine(onnx_path, precision="fp16", calibrator=None):
    builder = trt.Builder(TRT_LOGGER)
    network = builder.create_network(
        1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
    )
    parser = trt.OnnxParser(network, TRT_LOGGER)

    with open(onnx_path, "rb") as f:
        if not parser.parse(f.read()):
            for error in range(parser.num_errors):
                print(parser.get_error(error))
            raise RuntimeError("ONNX parse failed")

    config = builder.create_builder_config()
    config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30)  # 1 GB

    if precision == "fp16":
        config.set_flag(trt.BuilderFlag.FP16)
    elif precision == "int8":
        config.set_flag(trt.BuilderFlag.INT8)
        config.int8_calibrator = calibrator

    return builder.build_serialized_network(network, config)

The engine build process runs on the target GPU — you cannot build on a desktop and deploy on Jetson. The engine is GPU-architecture specific. Build it on the same hardware (or same architecture) you'll run inference on.

Build time is slow: a YOLOv8m engine takes 3–8 minutes on Jetson AGX Orin. This is expected. The engine is then saved to disk and loaded at inference time in milliseconds.

Jetson-Specific Considerations

Jetson devices use unified memory — CPU and GPU share the same physical RAM. This changes how you think about TensorRT engines:

No PCIe transfer overhead. On a desktop GPU, every inference call involves transferring input data from CPU RAM to GPU VRAM over PCIe (typically 16 GB/s). On Jetson, this cost is zero. This makes Jetson more competitive on batch size 1 (single-frame) inference than its raw compute numbers suggest.

Thermal throttling. Sustained high-throughput inference on Jetson heats the SoC. The Jetson will reduce clock speeds to stay within thermal limits. For production deployments, thermal management (heatsink, fan, power mode selection) directly affects sustained throughput.

NVDLA. Jetson Orin and AGX Xavier include NVDLA (Deep Learning Accelerator) cores. TensorRT can target these for supported layer types, running them in parallel with the GPU. For pure convolution workloads this can add 20–30% throughput, but not all model architectures benefit.

My Take

The single biggest mistake I see when people first add TensorRT to a project: they build the engine on a development machine with a different GPU than the deployment target. The engine builds fine, loads on the wrong hardware, and either crashes or produces garbage output. Always build on the target.

The second mistake: treating INT8 as free. It's faster, but requires calibration discipline. On one project we had 3% mAP drop from a poorly calibrated INT8 model that we didn't catch until production testing.

When it works correctly, TensorRT is genuinely transformative for edge deployment — the difference between a model running at 8 FPS and 45 FPS on the same hardware is the difference between a prototype and a product.

Reproduce on Your Hardware

# Export PyTorch model to ONNX
python -c "
import torch
model = torch.load('model.pt').eval()
dummy = torch.randn(1, 3, 640, 640)
torch.onnx.export(model, dummy, 'model.onnx',
    opset_version=17, input_names=['input'], output_names=['output'],
    dynamic_axes={'input': {0: 'batch'}, 'output': {0: 'batch'}})
"

# Build TensorRT engine (FP16)
trtexec --onnx=model.onnx --saveEngine=model_fp16.engine --fp16

# Benchmark
trtexec --loadEngine=model_fp16.engine --iterations=100

trtexec is included with TensorRT and is the fastest way to validate an export and get a throughput baseline before writing inference code.

Tested on Jetson AGX Orin (JetPack 6.x), Jetson AGX Thor (JetPack 6.8), and RTX 3090 (CUDA 12.4, TensorRT 10.x). Build and runtime behavior may differ across TensorRT major versions.