Qwen3.8-27B Benchmarked: 58 tok/s on an RTX 3090 at Q4, 4.1 tok/s on a Jetson Thor at BF16
Qwen3.8-27B sits at an awkward size: 51 GB of BF16 weights won't fit on a 24 GB RTX 3090, while a Jetson AGX Thor with 122 GB of unified memory fits it easily — and then runs it painfully slowly.
This post records what both machines actually do with the same model, and why a 14× gap can be predicted from two numbers.
Test Setup
| RTX 3090 workstation | Jetson AGX Thor | |
|---|---|---|
| GPU | GeForce RTX 3090 | NVIDIA Thor (sm110) |
| VRAM / unified memory | 24 GB GDDR6X | 122 GB LPDDR5X |
| Rated memory bandwidth | 936 GB/s | 273 GB/s |
| Driver | 580.173.02 | 580.00 / JetPack R38.4.0 |
| OS | Ubuntu 22.04.5 LTS | Linux 6.8.12-tegra |
| Serving stack | Ollama 0.32.14 | vLLM 0.25.2 (self-built for sm110) |
| Model format | GGUF Q4_K_M, 17 GB | Safetensors BF16, 51.1 GiB |
The model: Qwen3.8-27B, 27.3B dense parameters, qwen35 architecture (linear attention with a full-attention layer every 4th layer), 262K native context, a 460M vision tower, and thinking support.
Why the Two Sides Use Different Quantization
This isn't sloppiness — it's what the hardware allows, and it matters for reading the numbers below.
The 3090 can only run Q4. BF16 is 51 GB and 24 GB of VRAM cannot hold it. There is no alternative. Q4_K_M brings it to 17 GB, leaving 7 GB for KV cache.
Thor can only run BF16. I checked both Thor boxes: neither ollama manifest directory contains any qwen3.8 GGUF, and there's no separate dockerized ollama instance either. What is on the machine is the HuggingFace BF16 checkpoint (18 safetensors shards, 52 GB). Matching quantization would have meant downloading another 17 GB — and using the BF16 weights answers a more interesting question anyway: what is Thor's 122 GB of unified memory actually worth?
So this is a "best each machine can do" comparison, not a controlled experiment.
Results
Four scenarios, three runs each at concurrency 1 and two rounds at concurrency 4, temperature=0, 256 max output tokens.
Decode speed (tok/s, higher is better)
| Input length | Concurrency | RTX 3090 (Q4) | Thor (BF16) | Ratio |
|---|---|---|---|---|
| 136 tok | 1 | 58.2 | 4.1 | 14.2× |
| 760 tok | 1 | 46.1 | 4.1 | 11.2× |
| 2872 tok | 1 | 54.0 | 4.1 | 13.2× |
| 760 tok | 4 | 46.2 | 4.0 | 11.6× |
Time to first token (seconds, lower is better)
| Input length | Concurrency | RTX 3090 | Thor |
|---|---|---|---|
| 136 tok | 1 | 0.63 | 0.52 |
| 760 tok | 1 | 0.76 | 0.89 |
| 2872 tok | 1 | 1.30 | 2.86 |
| 760 tok | 4 | 2.79 | 3.21 |
Aggregate throughput (tok/s across all concurrent requests)
| Scenario | RTX 3090 | Thor |
|---|---|---|
| 760 tok, concurrency 1 | 26.9 | 4.0 |
| 760 tok, concurrency 4 | 34.5 (1.28×) | 15.1 (3.74×) |
Reading 1: Both Machines Sit on the Bandwidth Roofline
Autoregressive decoding reads every weight from memory once per generated token. So the ceiling is:
max decode speed = memory bandwidth ÷ weight size
Plugging in:
- Thor: 273 GB/s ÷ 54.9 GB ≈ 5.0 tok/s; measured 4.1, or 82% of theoretical
- 3090: 936 GB/s ÷ 17 GB ≈ 55 tok/s; measured 58.2 — at or slightly above the roofline
The 3090 exceeding 100% isn't a perpetual motion machine, it's an oversized denominator: those 17 GB include the 460M vision tower and clip projector, which text-only decoding never touches, and Ollama's eval_duration counter measures only the decode loop, excluding framework overhead. Excluding the vision parts at ~16.3 GB, the measurement implies roughly 949 GB/s — essentially the rated 936 GB/s.
The conclusion: neither machine is short on compute. Both are short on the ability to move weights. That also explains why Thor's 4.1 tok/s doesn't budge across input lengths — context grows from 136 to 2872 tokens and decode speed doesn't move at all, because the bottleneck was never attention, it's hauling those 51 GB of weights.
The gap factors cleanly:
bandwidth ratio 936 / 273 = 3.4×
weight ratio 54.9 / 17 = 3.2×
predicted gap 3.4 × 3.2 ≈ 11× measured 11–14×
In other words, roughly half the 14× comes from quantization (can you get to Q4?) and half from memory bandwidth (GDDR6X vs LPDDR5X). The remainder is vLLM running under --enforce-eager without CUDA Graphs, plus differing counter conventions on the two sides.
Reading 2: Thor Loses on Single Stream, Wins Back on Concurrency
On raw tok/s Thor looks hopeless. Push concurrency to 4 and the picture changes:
- The 3090 goes from 26.9 to 34.5 — only 1.28×
- Thor goes from 4.0 to 15.1 — 3.74×, nearly linear
The reason is again memory. Decoding is bandwidth-bound, but multiple requests can share a single pass over the weights: batch 4 requests together, read the weights once, produce 4 outputs, marginal cost near zero. Thor has 53 GB of KV cache space (vLLM reported a maximum concurrency of 79×), so the batch can grow as far as you want.
The 3090 can't scale because 17 GB of weights plus KV cache already fills its 24 GB, and Ollama's parallelism is capped in that configuration — the 4 requests largely queue.
Which makes these machines suited to genuinely different jobs:
- The 3090 is for one interactive user: chat, coding assistance, where 58 tok/s reads as fluent
- Thor is for multiple background streams: 4.1 tok/s is too slow for a human to watch, but 15 tok/s aggregate across 4 async jobs (bulk labelling, log analysis, multi-camera understanding) is usable — and it holds models the 3090 simply cannot
Reading 3: Prefill Is a Separate Ledger
Prefill is compute-bound rather than bandwidth-bound, so it follows different rules. Thor prefills the 2872-token input at roughly 1000 tok/s for a 2.86 s TTFT — slower than the 3090, but nowhere near 14× slower. On short inputs Thor's TTFT (0.52 s) actually beats the 3090's (0.63 s).
One trap worth flagging: Ollama's reported prefill numbers can't be taken at face value. My harness repeats the same prompt three times; from the second run on it hits the prefix cache, prompt_eval_duration only counts newly computed tokens, and a 2832-token prefill gets reported as 11,227 tok/s. That number is fiction. The 2.5× spread between minimum TTFT (0.51 s) and mean (1.30 s) is the cache hit showing through. vLLM has prefix caching on by default too, but its TTFT held steady at 2.83–2.86 s across all three runs, showing no cache benefit.
Three Traps I Hit
1. On Thor, "RAM" and "VRAM" are the same pool, and vLLM's startup check will stop you.
My first launch used the habitual --gpu-memory-utilization 0.72 and was refused outright:
ValueError: Free memory on device cuda:0 (68.15/122.82 GiB) on startup is
less than desired GPU memory utilization (0.72, 88.43 GiB).
On Thor that fraction is computed against the full 122 GB, not against what's free. Ollama was holding a 44 GB resident model at the time, leaving only 68 GB. Unloading every ollama model and dropping the setting to 0.53 got it running.
2. Unified-memory allocations leak far worse than discrete VRAM.
After the benchmark I tore the vLLM container down and 110 GB never came back — total process RSS across the machine was 2 GB, nvidia-smi showed no compute processes, no orphaned containerd shims, yet free reported 111 GB used. And this isn't a force-kill artifact: a separate SGLang container that exited cleanly via docker stop left 85 GB stranded the same way. On a discrete GPU, VRAM returns when the process dies. On Thor a leaked CUDA allocation eats system memory until you reboot the box.
3. Don't assume "it's installed" means "it runs."
I started out believing all four machines (rog3090, giga3090, thor, thor-jd) had this model ready. What I actually found: only rog3090's ollama had qwen3.8:27b; giga3090 had no ollama at all and its GPU was busy with another 15.8 GB job; neither Thor had the model in ollama, and only one had the HF BF16 weights. One machine out of four could start immediately.
How to Choose
| Your situation | Recommendation |
|---|---|
| Single local assistant that must keep up with typing | RTX 3090 + Q4. 58 tok/s, cheap and sufficient |
| Model too large for 24 GB | Thor. Slow beats not running at all |
| Multiple async background jobs | Thor + vLLM, run a real batch and harvest aggregate throughput |
| Large model and fast | Neither of these. Look at the A100/H100 tier |
If you must chase single-stream speed on Thor, the only lever that works is shrinking the weights: FP8 or NVFP4 takes them to roughly 25 GB, and the roofline then permits 8–10 tok/s. In a bandwidth-bound regime that's the whole game — kernel optimization buys nothing when the compute units were never saturated.
Methodology
Both sides used streaming APIs, with TTFT recorded client-side and decode speed taken from server-side counters (Ollama's eval_count/eval_duration; vLLM's completion_tokens from the usage field combined with stream timestamps). Each scenario ran one warmup, then three runs at concurrency 1 and two rounds at concurrency 4, with temperature=0 and max_tokens=256.
One convention difference to note: the vLLM side emitted thinking blocks (163–196 tokens average) while the Ollama side had thinking disabled (44–49 tokens average). That changes total output length but not per-token decode speed, and per-token decode speed is what this post compares.