Model serving · evidence record 2026-08-27

What does everyone get wrong about optimizing Qwen3.8-27B on H100, H200, B200, GB300, RTX PRO 6000, and RTX 5090 with vLLM?

A source-grounded optimization dossier spanning H100, H200, B200, GB300 NVL72, RTX PRO 6000 Blackwell, and RTX 5090, with workload-specific vLLM starting points, hardware identities, memory and context boundaries, MTP measurement rules, and a reproducible benchmark matrix.

The useful answer is not one launch command. It is a versioned workload envelope: exact checkpoint, hardware identity and topology, context and output distributions, concurrency, latency objective, reasoning mode, tool use, modality, and measured acceptance behavior for MTP.

01 / SOURCE-LINKED BASELINE

What is actually established

These are compatibility and architecture facts from Qwen, vLLM, and NVIDIA. They establish the benchmark boundary; they do not establish a universal optimum.

Checkpoint

Qwen/Qwen3.8-27B

Official open-weight release dated 2026-08-14. Use the canonical model ID when comparing serving results.

Qwen · Qwen · Hugging Face

Architecture

27B dense · 64 layers

48 Gated DeltaNet linear-attention layers and 16 full-attention layers, plus a trained MTP head and vision encoder.

Qwen · Hugging Face

Context

262,144 native

The official card documents extension to 1,000,000 tokens with static YaRN, while warning that the scaling factor can affect shorter inputs.

Qwen · Hugging Face

Serving engine

vLLM 0.17.0+

The official recipe establishes compatibility. Its published Blackwell examples are not single-H200 benchmarks.

vLLM

H200 envelope

141 GB HBM3e · 4.8 TB/s

Record the SXM or NVL SKU in every result because compute, power, and interconnect characteristics differ.

NVIDIA

Correctness flags

qwen3 · qwen3_coder

Reasoning extraction and automatic tool calls require the matching reasoning and tool-call parsers in the official serving example.

Qwen

HARDWARE IDENTITY LEDGER

Six names, six different deployment envelopes

Memory capacity is only the first boundary. Edition, topology, interconnect, runtime, precision, and workload still have to travel with every result.

NVIDIA H100

H100 SXM 80GB / H100 NVL 94GB

80 GB at 3.35 TB/s / 94 GB at 3.9 TB/s

H100 is not one interchangeable SKU. Pin SXM or NVL, GPU count, tensor parallelism, and interconnect before comparing capacity or latency.

NVIDIA

NVIDIA H200

H200 SXM or H200 NVL

141 GB HBM3e · 4.8 TB/s

The benchmark commands below are conservative TP1 H200 starting points. NVIDIA specifications do not establish Qwen throughput or a vLLM optimum.

NVIDIA

NVIDIA B200

B200

180 GB HBM3e · 8 TB/s

Blackwell capacity and bandwidth do not make an NVFP4 recipe portable to Hopper or prove the best batching, cache, or MTP configuration.

NVIDIA · vLLM

NVIDIA GB300 NVL72

72× Blackwell Ultra GPUs

288 GB per GPU · 20.7 TB aggregate HBM3e

This is a rack-scale topology. A published TP4 launch shape is a compatibility point inside that system, not an independently measured production optimum.

NVIDIA · vLLM

NVIDIA RTX PRO 6000 Blackwell

Server Edition / Workstation Edition

96 GB GDDR7 · 1,597 or 1,792 GB/s

Name the edition: equal capacity does not mean equal bandwidth, power, cooling, interconnect, or multi-GPU behavior.

NVIDIA · NVIDIA

NVIDIA GeForce RTX 5090

RTX 5090

32 GB GDDR7

Official vLLM recipe observations cover specific quantized checkpoints and runtime builds. They are startup and cache-capacity evidence, not general throughput claims.

NVIDIA · vLLM

02 / BENCHMARK STARTING POINTS

Three workloads, three starting profiles

These commands remain falsifiable starting points for one H200—not claimed production winners or templates to copy across the hardware ledger. The sweep and acceptance criteria beside each command are part of the configuration.

Latency first

Interactive coding agents

Mostly 8K–32K prompts, tool calls enabled, moderate concurrency, p95 first-token latency prioritized.

STARTING COMMANDTP1 · BF16 BASELINE

vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Measure before promotion

p50/p95 TTFT
p50/p95 TPOT and ITL
tool-call parse success
prefix-cache hit rate

This is a conservative benchmark baseline, not a measured optimum. Sweep sequence and batched-token limits against the real prompt distribution.

Throughput first

Shared enterprise serving

Many short-to-medium requests, bounded outputs, queueing allowed, aggregate tokens per second prioritized.

STARTING COMMANDTP1 · BF16 BASELINE

vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 32768 \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Measure before promotion

request and token throughput
queue time and p95 E2E latency
MTP acceptance rate and length
GPU memory headroom

MTP is a variable, not a free speedup. Compare it disabled and at 1/2/3 speculative tokens under the same arrival and output distributions.

Context first

Repository and document analysis

128K–262K prompts, low concurrency, large prefix reuse, context capacity prioritized over request density.

STARTING COMMANDTP1 · BF16 BASELINE

vllm serve Qwen/Qwen3.8-27B \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder

Measure before promotion

prefill latency by prompt length
peak cache memory
prefix-cache reuse
quality at the target context length

Validate capacity before load testing. The hybrid full-attention and recurrent-state caches make simple KV-tokens-per-GB estimates incomplete.

03 / DECISION MATRIX

Every lever moves a different metric

Record the full workload distribution. A flag that wins a short-prompt throughput test can lose on long-context TTFT, tail latency, memory headroom, or semantic correctness.

Lever	What it can favor	What it can cost	Evidence rule
`max_model_len`	Memory headroom and concurrency when reduced	Context capacity	Native 262K does not imply every deployment should reserve 262K.
`max_num_batched_tokens`	TTFT/throughput when raised	Inter-token latency at aggressive values	vLLM documents this as a workload-dependent scheduling tradeoff.
`max_num_seqs`	Concurrent request density when raised	Per-request cache headroom and tail latency	Sweep with a fixed request distribution and record queue time.
`MTP tokens`	Decode latency in memory-bound low/medium-QPS regimes	Draft overhead and poor acceptance on mismatched workloads	Report acceptance rate, accepted length, TPOT, and ITL together.
`reasoning effort`	Task quality and longer deliberation	Output tokens, latency, and concurrent capacity	Keep sampling and output distributions fixed across comparisons.
`vision/video`	Multimodal task coverage	Encoder work, media tokens, and processor cache	Benchmark separately from text-only serving.

CONTEXT EXTENSION

One million tokens is a separate deployment profile.

Qwen documents a static YaRN factor of 4 for 1M context and warns that static scaling can affect shorter inputs. Choose the factor for the typical workload, then benchmark quality and memory at that length.

OFFICIAL YARN FRAGMENTNOT A CAPACITY CLAIM

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-27B \
  --hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}' \
  --max-model-len 1000000

04 / REPRODUCIBLE MEASUREMENT

The benchmark packet

A result is comparable only when the model, runtime, hardware, workload, and metric definitions travel with the number.

01Pin model revision, vLLM, CUDA, PyTorch, attention backend, exact GPU edition, count, topology, and interconnect.
02Preserve prompt/output distributions, context buckets, modality, reasoning effort, and tool-call rate.
03Report p50/p95 TTFT, TPOT, ITL, E2E latency, request/token throughput, queue time, and peak memory.
04For MTP, add acceptance rate and accepted length; compare disabled and 1/2/3-token configurations.

Claims this record rejects

Copying Blackwell NVFP4 numbers into a Hopper capacity claim.
Treating the official TP4 recipe as evidence for one-GPU performance.
Treating H100 SXM and NVL or RTX PRO 6000 Server and Workstation editions as interchangeable.
Enabling 1M static YaRN for predominantly short prompts.
Comparing reasoning and non-reasoning runs with different output distributions.
Calling MTP faster from TPOT alone while ignoring ITL and acceptance rate.
Omitting reasoning or tool parsers and benchmarking a semantically broken API.
Mixing text-only, image, and video requests in one unexplained throughput number.

SEARCH COVERAGE

Questions this dossier is built to resolve

Qwen3.8-27B on one H200Qwen3.8-27B H100 versus H200Qwen3.8-27B B200 or GB300 vLLMQwen3.8-27B RTX PRO 6000Qwen3.8-27B RTX 5090Qwen3.8-27B vLLM configurationQwen3.8-27B H200 VRAMQwen3.8-27B H200 throughputQwen3.8-27B max-num-seqsQwen3.8-27B FP8 KV cacheQwen3.8-27B MTP vLLMQwen3.8-27B 262K contextQwen3.8 reasoning parser vLLMQwen3.8 tool calling vLLM

05 / PRIMARY SOURCES

Source ledger

Product identifiers and serving flags drift. Every claim above is tied to the source that defines its current boundary.

01
Qwen
Qwen3.8 official repository
Release identity, canonical checkpoint, vLLM parser flags, and serving examples.
02
Qwen · Hugging Face
Qwen3.8-27B model card
Architecture, native context, YaRN extension, reasoning controls, sampling guidance, and multimodal behavior.
03
vLLM
Qwen3.8-27B vLLM recipe
Supported vLLM release, verified serving surface, and MTP configuration fragment.
04
vLLM
vLLM optimization and tuning
Chunked-prefill, batched-token, sequence-count, and latency-versus-throughput tuning semantics.
05
vLLM
vLLM speculative decoding
MTP workload boundary, acceptance metrics, and interpretation of inter-token latency under speculation.
06
vLLM
vLLM serving benchmark CLI
TTFT, TPOT, ITL, request throughput, token throughput, and controlled-concurrency methodology.
07
NVIDIA
NVIDIA H200 Tensor Core GPU
H200 memory capacity, bandwidth, and the distinction between SXM and NVL variants.
08
NVIDIA
NVIDIA H100 Tensor Core GPU
H100 SXM and H100 NVL memory capacities and bandwidth envelopes.
09
NVIDIA
NVIDIA HGX B200 components
B200 HBM3e capacity, bandwidth, and Blackwell platform identity.
10
NVIDIA
NVIDIA GB300 NVL72
GB300 NVL72 rack-scale GPU count, per-GPU memory, NVLink fabric, and aggregate memory envelope.
11
NVIDIA
NVIDIA RTX PRO 6000 Blackwell Server Edition
The 96 GB Server Edition identity and memory bandwidth.
12
NVIDIA
NVIDIA RTX PRO 6000 Blackwell Workstation Edition data sheet
The 96 GB Workstation Edition identity, PCIe generation, and memory bandwidth.
13
NVIDIA
NVIDIA GeForce RTX 5090
RTX 5090 device identity and its 32 GB GDDR7 memory envelope.

WORKLOAD-SPECIFIC EVIDENCE

Ask for the configuration your workload actually needs.

Include GPU SKU and count, concurrency, prompt and output distributions, latency and throughput targets, context, precision, reasoning, tool-calling, and vision requirements. Send only non-sensitive workload context.

“Compare H200, B200, and RTX PRO 6000 configurations for 40 enterprise coding agents with 16K prompts, 4K outputs, tool calling, and p95 TTFT below two seconds.”

Read the task llms.txt