Checking vLLM on Thor: Qwen3-0.6B, 8 of 9 Probes, and Three API Demos
On November 9, 2026, I ran the quick profile of my thortest harness on Jetson AGX Thor. The vLLM × Qwen3-0.6B combination completed its functional and performance subtests: eight of nine functional probes passed, while JSON Schema output failed. The overall matrix remained incomplete.
There are no completed SGLang, 8B Dense or 30B-A3B MoE results. The other quick-profile model, Granite, failed during download; that is not evidence of an inference-engine compatibility failure.
Conditions and evidence
The machine was a marketed 128 GB Thor, sm_110, L4T R39.2, host CUDA driver API 13.2, MAXN. vLLM used the pinned official nightly-aarch64 image, version 0.33.0rc1.dev806+ga2bcb5657. Qwen3-0.6B ran in BF16 with an 8,192-token limit and up to four active sequences.
Flash-Next was still resident. Only about 18.3 GB was available at the start, so these numbers describe a shared machine rather than an isolated maximum-performance run. The saved result marks the combination partial.
I checked thortest/reports/20261109-051803/results.json, lib/bench.py and lib/probes.py. Download the selected original report fields, with probe details and all six throughput rows. Machine addresses and unrelated services are omitted. See the large-model deployment article for the co-resident workload.
Functional probes
| Probe | Observed result |
|---|---|
| Model list | Pass: expected served ID |
| Arithmetic | Pass: 17 × 23 = 391 |
| Chinese | Pass: Beijing answer |
| SSE streaming | Pass: 35 content chunks |
| JSON Schema | Fail: repeated digits and incomplete JSON |
| Function calling | Pass: parsed get_weather({"city":"Paris"}) |
| Reasoning separation | Pass: separate reasoning and final answer |
| Context retrieval | Pass: found 7429 in a 5,769-token prompt |
| Four concurrent requests | Pass: all returned nonempty answers |
“Pass” means the harness criterion passed. The tool probe did not call a weather service, and the concurrency probe checked for nonempty responses. These are not full Agent workflow or production reliability assessments.
The failed JSON output kept appending digits in population_million and did not close the object within its 128-token budget. Token budget, constrained decoding and model behavior need further investigation. This one probe cannot establish a general failure rate or isolate the engine as the cause.
Throughput definitions
The synthetic workloads targeted 512→256 and 4096→256 tokens. Actual mean input counts are saved in the download. Each concurrency level used 3, 4 or 8 requests, with ignore_eos to maintain the output budget.
| Input | Concurrency | Requests | Aggregate output tok/s | Mean TTFT |
|---|---|---|---|---|
| Approx. 512 | 1 | 3 | 142.25 | 43.3 ms |
| Approx. 512 | 2 | 4 | 269.77 | 60.5 ms |
| Approx. 512 | 4 | 8 | 493.76 | 89.9 ms |
| Approx. 4096 | 1 | 3 | 111.85 | 147.6 ms |
| Approx. 4096 | 2 | 4 | 176.48 | 247.4 ms |
| Approx. 4096 | 4 | 8 | 243.24 | 450.2 ms |
Aggregate output throughput is the total completion-token count divided by whole-round wall time. The four-stream 494 tok/s number is not the speed of each stream. The report separately computes per-stream speed as the inverse of median TPOT: about 147.78 tok/s at concurrency one and 128.63 tok/s at concurrency four for the short workload.
The prefill probe reported roughly 29,790 and 25,441 tok/s at actual average inputs of 2,034 and 6,084 tokens. It estimates prompt tokens / TTFT, including API and first-token overhead, rather than timing an isolated prefill GPU kernel. There were just two requests per length. These small samples do not support stable tail-latency or sustained-capacity claims.
Three downloadable replay demos
Download demos.py. It uses the Python standard library; --dry-run prints the request body. These examples were prepared from the verified API shapes and were not rerun against a model during article publication. They are not additional Thor measurements.
A service must already be running. Use loopback on the Thor, or an SSH local port forward from another computer. Check /v1/models for the actual served name.
Demo 1: streaming with separate reasoning
python3 demos.py chat --model qwen38-flash-next
python3 demos.py chat --model qwen38-flash-next --thinking --max-tokens 2048
The prompt asks for a three-sentence MoE explanation. Final content goes to stdout; reasoning and timing go to stderr. The script reports first output, first final-answer content, elapsed time and finish reason. A length-limited response is flagged instead of being counted as normally completed.
Demo 2: parsing a tool call
python3 demos.py tool --model qwen3-0.6b \
--base-url http://127.0.0.1:8100/v1
The client verifies the get_weather name, JSON arguments and Paris city parameter. It executes no external tool and supplies no invented live weather. The historical 0.6B probe passed this request shape; execution, tool-result feedback and final response still require a separate workflow test.
Demo 3: replaying JSON Schema output
python3 demos.py schema --model qwen3-0.6b \
--base-url http://127.0.0.1:8100/v1 --max-tokens 128
This requests Tokyo facts with city, country and population_million. The client checks complete JSON, exact fields, basic types and normal completion, not factual accuracy. The historical probe failed on this shape. Save the full response, finish reason and model revision when replaying it.
Thinking is disabled by default in these demos; the server must support the corresponding chat-template option and parsers. Download the large-model launcher from the deployment article. The 0.6B service on port 8100 was a historical test instance and must be started separately. The website does not provide public access to the lab inference server.
Interruption and remaining work
The machine became unreachable during the SGLang image pull while Flash-Next was resident. The saved evidence establishes low memory headroom, not a unique root cause. An interrupted image download is not a failed SGLang inference test.
The next run should prepare downloads first and test each engine on an otherwise idle machine. The lab harness provides:
cd ~/thorllmtest/thortest
./run.sh --profile standard --stop-running
This stops existing model containers identified by the harness and attempts to restore them after the run, so schedule a maintenance window. Completed SGLang tests, 8B / 30B-A3B results, a JSON failure reproduction, larger samples and sustained runs remain outstanding.