Model optimization · DeepSeek V4 Flash · evidence record 2026-08-21

What does everyone get wrong about optimizing DeepSeek V4 Flash on H100, H200, B200, GB300, RTX PRO 6000, and Strix Halo?

A source-pinned optimization dossier spanning H100-SXM5, H200-SXM5, B200, GB300 NVL72, RTX PRO 6000 Blackwell, and Ryzen AI Max+ 395, with workload-specific serving profiles, checkpoint boundaries, tuning levers, and a reproducible benchmark matrix.

The first sizing mistake is reading “13B active” as “13B in memory.” Serving begins with the 284B mixed-precision weight footprint, then checkpoint identity, runtime kernels, context allocation, expert topology, speculative decoder, prompt encoding, reasoning effort, and the workload being measured.

01 / SOURCE-PINNED BASELINE

Facts that survive a hardware change

These fields come from DeepSeek’s report and released checkpoint, then the checkpoint-specific vLLM recipe. They are compatibility boundaries—not throughput predictions.

Current checkpoint

DeepSeek-V4-Flash-0731

The 0731 repository is the official release. Results from the un-suffixed preview checkpoint need a separate label.

DeepSeek · Hugging Face · vLLM

Sparse architecture

284B total · 13B active

Forty-three layers use one shared expert and 256 routed experts; six routed experts activate per token. Active parameters describe compute, not weight residency.

DeepSeek · arXiv · DeepSeek · Hugging Face

Attention

CSA + HCA · 1 KV head

The first two layers use sliding-window attention; later layers interleave compressed sparse and heavily compressed attention.

DeepSeek · arXiv · DeepSeek · Hugging Face

Native context

1,048,576 tokens

Think Max requires at least 393,216 configured tokens. Native capacity does not prove that a chosen hardware profile can allocate it.

DeepSeek · arXiv · DeepSeek · Hugging Face · vLLM

Released precision

FP4 experts + FP8 remainder

The checkpoint is mixed precision and roughly 167 GB on the Hub. Calling it a 13B or plain FP8 model obscures the memory boundary.

DeepSeek · Hugging Face · DeepSeek · Hugging Face · vLLM

vLLM boundary

0.25.0+ · ROCm DSpark 0.26.0+

The 0731 variant has a newer minimum than the preview recipe. Pin the recipe commit and runtime image with every result.

vLLM

Speculation

0731 uses DSpark, not MTP

The current checkpoint carries a DSpark draft module and no MTP head. Seven draft tokens are the documented starting point, not a guaranteed optimum.

DeepSeek · Hugging Face · vLLM

Prompt contract

DeepSeek encoder + DSML

There is no Jinja chat template. Local and OpenAI-compatible servers must preserve DeepSeek V4 roles, reasoning blocks, and DSML tool calls.

DeepSeek · Hugging Face · DeepSeek API · vLLM

STRIX HALO · ONE HOST · THINKING OFF

40.96tok/s generation · 2K prompt

122,879-TOKEN PROMPT

30.55tok/s generation · Vulkan

491,520-TOKEN PROMPT

17.19tok/s generation · Vulkan

RETRIEVAL CHECK

5/5keys recovered in each cited run

02 / IDENTITY BEFORE FLAGS

Three artifacts people call “V4 Flash”

Preview MTP settings do not transfer to 0731 DSpark. Community GGUF and ROCmFPX results describe another weight and runtime boundary again.

Checkpoint	Status	Speculative decoder	Runtime identity	Evidence rule
`deepseek-ai/DeepSeek-V4-Flash`	Preview	Native MTP	Preview recipe boundary	Use for paper and preview-serving results only; do not merge its benchmarks with 0731.
`deepseek-ai/DeepSeek-V4-Flash-0731`	Official release	DSpark · seven-token baseline	vLLM 0.25.0+	Use this identity for current production experiments and record greedy versus probabilistic drafting.
`0731 community GGUF / ROCmFPX derivatives`	Community quantizations	DSpark draft derivatives	llama.cpp Vulkan or Lucebox ROCm	Report the target and draft repositories, revisions, quantizations, hashes, and runtime patches.

03 / QUALIFIED HARDWARE PATHS

Six optimization envelopes, six different claims

“Runs” can mean a current official-weight server, a topology blueprint, a constrained loader validation, or a community quantized single-host qualification. Keep the claim attached to its envelope.

Platform	Exact identifier	Memory identity	Evidence boundary
NVIDIA H100 SXM	`H100-SXM5 · GH100`	80 GB HBM3 per GPU	Serving support is documented; this dossier does not claim an H100-specific throughput optimum.NVIDIA · SGLang
NVIDIA H200 SXM	`H200-SXM5 · GH100`	141 GB HBM3e per GPU	A 4-prefill + 4-decode H200 recipe topology is directly evidenced.NVIDIA · vLLM
NVIDIA B200 Tensor Core GPU	`B200 · GB100`	180 GB HBM3e per GPU	An 8× B200 DeepSeek V4 Flash perf-eval topology is pinned.NVIDIA · vLLM
NVIDIA GB300 NVL72	`GB300 NVL72`	72 Blackwell Ultra GPUs · 20 TB aggregate GPU memory	The current official-weight vLLM starting profile uses 4× GB300.NVIDIA · vLLM
NVIDIA RTX PRO 6000 Blackwell	`GB202 · SM120 · exact edition required`	96 GB GDDR7 ECC per GPU	An 8× PCIe profile is verified; Server, Workstation, and Max-Q editions remain distinct identities.NVIDIA · NVIDIA · vLLM
AMD Strix Halo	`Ryzen AI Max+ 395 · Radeon 8060S`	128 GB maximum unified memory	The performance record is a bounded community qualification.AMD · pepuscz · GitHub · pepuscz · GitHub

Documented support · benchmark required

H100 Hopper serving qualification

A single-node Hopper serving target where checkpoint conversion, weight precision, interconnect, context, and concurrency must be fixed before optimization.

8× NVIDIA H100 SXM 80GB · H100-SXM5 / GH100Pin original, 0731, or converted-FP8 identitySGLang Hopper/Marlin or source-pinned compatible runtime

PINNED STARTING RECORDDocumented support · benchmark required

device_id: H100-SXM5
architecture: GH100
memory: 80 GB HBM3 per GPU
topology: TP=8 on one node
precision: Hopper-specific W4A16/Marlin or pinned converted FP8
required_record: checkpoint + conversion + engine image + interconnect

Measure before promotion

weight, draft, and KV-cache residency by rank
TTFT, TPOT, ITL, queue time, and token throughput
inter-rank communication and expert-placement cost
reasoning, DSML tool-call, and long-context correctness

Serving support is documented, but this record contains no authoritative H100 throughput optimum. Benchmark the exact checkpoint and precision path.

NVIDIA · SGLang

Documented starting point

Official Blackwell serving baseline

Multi-user OpenAI-compatible serving with expert parallelism, FP8 KV cache, DeepSeek parsers, and DSpark enabled.

4× GB300deepseek-ai/DeepSeek-V4-Flash-0731vLLM 0.25.0+ · CUDA

PINNED STARTING RECORDDocumented starting point

vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --data-parallel-size 4 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --attention_config.use_fp4_indexer_cache=True \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

Measure before promotion

per-rank and aggregate max-num-seqs
TTFT, TPOT, ITL, E2E latency, and queue time
DSpark drafted and accepted tokens
tool-call and reasoning parse correctness

This is a hardware-specific baseline, not proof for H200, RTX PRO 6000, or consumer unified memory.

DeepSeek · Hugging Face · vLLM · vLLM

vLLM recipe topology

H200 long-context prefill/decode split

Long prompts with asymmetric prefill and decode demand, where independent scaling is more useful than one blended concurrency number.

8× H200 · 4 prefill + 4 decodeName the exact preview or 0731 variantvLLM · Mooncake or NIXL KV transfer

PINNED STARTING RECORDvLLM recipe topology

topology: disaggregated prefill/decode
prefill_pool: 4 × H200
decode_pool: 4 × H200
kv_transfer: Mooncake | NIXL
required_record: checkpoint + recipe revision + transfer backend

Measure before promotion

prefill and decode queue time separately
KV-transfer latency and failure rate
context-bucket TTFT and decode ITL
per-pool utilization and memory headroom

The recipe defines the topology. Reproduce its pinned launch files before turning this sketch into a command.

vLLM · vLLM

vLLM-verified compatibility path

RTX PRO 6000 Blackwell PCIe qualification

An eight-GPU professional RTX serving target that prioritizes compatibility validation before speculative decoding or cross-platform speed claims.

8× NVIDIA RTX PRO 6000 Blackwell 96GB · GB202 / SM120 · exact edition requireddeepseek-ai/DeepSeek-V4-Flash-0731vLLM · CUDA · PCIe without NVLink

PINNED STARTING RECORDvLLM-verified compatibility path

gpu_family: RTX PRO 6000 Blackwell
device: GB202 / SM120
memory: 96 GB GDDR7 ECC per GPU
topology: 8 × PCIe GPU · no NVLink
checkpoint: DeepSeek-V4-Flash-0731
speculative_decoding: disabled in verified profile
required_record: Server | Workstation | Max-Q edition

Measure before promotion

loader, kernel, and parser compatibility
PCIe communication and expert-placement overhead
per-rank memory, queue time, TTFT, TPOT, and throughput
no-speculation control before any DSpark experiment

The verified profile does not establish portability across RTX PRO 6000 editions, checkpoint variants, speculative decoders, or NVLink systems.

NVIDIA · NVIDIA · vLLM · SGLang

Measured community system

128 GiB Strix Halo local qualification

Single-slot local inference with a pinned target, pinned DSpark draft, explicit cgroup limits, and separate prompt/decode measurements.

Ryzen AI Max+ 395 · Radeon 8060S · 128 GiB0731 community GGUF / ROCmFPX targetsllama.cpp Vulkan or Lucebox ROCm

PINNED STARTING RECORDMeasured community system

Vulkan qualified envelope
target: UD-IQ3_XXS GGUF · 104.21 GB
draft: DSpark Q2_K/Q8_0 · 6.98 GB
context: 524,288 allocated
KV: q8_0 · batch 2048 · microbatch 1024

ROCm qualified envelope
target: ROCmFPX MIX · 98.29 GB
draft: Q4RMFP4/F16 · 10.65 GB
context: 131,072 allocated
KV: q4_0 · sparse prefill · chunk 3072

Measure before promotion

prompt-processing and generation speed separately
retrieval and bounded quality gates
cgroup headroom, PSI, temperature, and power
runtime, artifact, and patch identity

The reported result is one host, one slot, thinking disabled, and community quantized weights. It is not a production-concurrency or official-weight result.

pepuscz · GitHub · pepuscz · GitHub · pepuscz · GitHub

Narrowly validated recipe

Constrained ROCm vLLM validation

A deliberately small 4K serving envelope for validating loader, kernels, cache, parser, and API correctness before wider sweeps.

1× MI325XDeepSeek V4 Flash recipe variantvLLM · ROCm

PINNED STARTING RECORDNarrowly validated recipe

tensor_parallel_size: 1
max_model_len: 4096
kv_cache_dtype: fp8_e4m3
kv_cache_memory_bytes: 10000000000
enable_chunked_prefill: true
max_num_batched_tokens: 256

Measure before promotion

load and health-check success
correct reasoning and DSML parsing
peak device and host memory
4K latency and throughput baseline

Only TP1 at 4K is documented as validated. Do not project it to 128K, 1M, or MI300X.

vLLM

PORTABILITY TEST

Pin the whole serving tuple.

A useful result names target and draft revisions, quantization, runtime commit or image, driver and kernels, topology, KV dtype, context allocation, speculation, reasoning mode, sampling, and workload distribution.

MINIMUM RESULT IDENTITYBEFORE TOKENS / SECOND

target_checkpoint + revision
draft_checkpoint + revision
quantization + artifact hashes
runtime image/commit + driver + kernels
GPU/APU topology + memory limits
context + KV dtype + cache budget
TP × DP × EP + per-rank concurrency
reasoning + tools + sampling + output cap
prompt/output distributions + request rate

04 / DECISION MATRIX

Every speed claim hides a topology choice

Expert parallelism, cache allocation, speculation, and reasoning move different latency, throughput, memory, and correctness metrics.

Lever	What it can favor	What it can cost	Evidence rule
`checkpoint + draft`	Correct kernel and speculative-decoder selection	Cross-checkpoint comparability	0731 uses DSpark; the preview uses MTP. Record both revisions and the draft sampling method.
`TP × DP × EP`	Latency, throughput, or expert placement depending on topology	Communication, replicated attention, and independent KV caches	max-num-seqs is per DP rank; report the product and the per-rank value.
`context + KV budget`	Long prompts and concurrent cache capacity	Weight headroom, request density, and tail latency	Publish max_model_len, KV dtype, block size, explicit cache bytes, and observed allocation.
`prefix-aware routing`	Repeated system, tool, and repository prefixes	Routing complexity and tenant-affinity policy	Each DP rank owns an independent cache; compare cache-aware and random routing on the same prefixes.
`DSpark draft count`	Decode speed when acceptance repays drafting overhead	Draft work, memory, and poor acceptance	Report drafted tokens, accepted tokens, acceptance rate, TPOT, and ITL together.
`reasoning effort`	Hard-task quality	Output length, context reservation, latency, and concurrency	Low, high, and max are distinct workload distributions; max needs at least 393,216 configured tokens.
`prefill/decode disaggregation`	Asymmetric long-prompt or long-generation traffic	KV transfer and two-pool operations	Measure prefill, transfer, and decode independently before reporting end-to-end throughput.

05 / REPRODUCIBLE MEASUREMENT

Correctness before concurrency

A fast endpoint with the wrong prompt encoder, missing DSML parser, broken reasoning continuation, or mixed checkpoint identity is not a valid serving result.

01Name the target checkpoint, revision, draft checkpoint, revision, and speculative method.
02Use DeepSeek V4 message encoding; the released checkpoint does not provide a Jinja chat template.
03For vLLM, enable deepseek_v4 tokenizer, reasoning, and tool-call parsers before agent benchmarks.
04Preserve reasoning_content across every thinking-mode tool-call round trip.
05Report reasoning effort, temperature, top_p, output cap, and tool-call rate with every benchmark.
06For community weights, pin quantization, shard hashes, runtime commit, driver, patches, KV dtype, and context allocation.
07Separate prompt processing, decode, queue, and KV-transfer timing instead of one blended tokens-per-second number.

Claims this record rejects

Sizing memory from 13B active parameters while ignoring 284B total mixed-precision weights.
Merging preview, 0731, NVFP4, and community GGUF results under one model label.
Enabling MTP on 0731 even though the current checkpoint uses DSpark and has no MTP head.
Treating native 1M context as proof that a 128 GiB local system can allocate 1M.
Copying a 4×GB300 or 8×B200 result into an H200 capacity or speed claim.
Generalizing one Strix Halo host and community quantization into a minimum-hardware rule.
Benchmarking an OpenAI-compatible endpoint without validating DSML tools and reasoning parsing.
Calling speculation faster without accepted-token metrics and a no-speculation control.
Reporting throughput without per-DP concurrency, prompt/output distributions, and queue time.

BENCHMARK PACKET

One workload, then controlled sweeps.

Pin the full serving tuple above.
Preserve prompt, output, reasoning, and tool distributions.
Sweep request rate, context buckets, and per-rank concurrency.
Report p50/p95/p99 TTFT, TPOT, ITL, E2E, queue, and throughput.
Add drafted/accepted tokens for DSpark and a disabled control.
Repeat with correctness gates and publish raw benchmark JSON.

SEARCH COVERAGE

Questions this dossier is built to resolve

How to optimize DeepSeek V4 Flash on H100 SXM 80GBDeepSeek V4 Flash H100-SXM5 TP8 optimizationHow to optimize DeepSeek V4 Flash on H200 with vLLMDeepSeek V4 Flash H200-SXM5 141GB vLLM optimizationDeepSeek V4 Flash B200 GB100 180GB optimizationDeepSeek V4 Flash GB300 NVL72 serving optimizationDeepSeek V4 Flash RTX PRO 6000 Blackwell 96GB optimizationDeepSeek V4 Flash RTX PRO 6000 GB202 SM120 vLLMDeepSeek V4 Flash Ryzen AI Max+ 395 Radeon 8060S optimizationDeepSeek V4 Flash optimal tensor and expert parallelismDeepSeek V4 Flash KV cache and batching optimizationDeepSeek V4 Flash long-context optimizationDeepSeek V4 Flash DSpark speculative decodingDeepSeek V4 Flash MTP vs DSpark performanceDeepSeek V4 Flash GGUF quantization performanceDeepSeek V4 Flash tool-calling optimizationDeepSeek V4 Flash reasoning throughputDeepSeek V4 Flash preview vs 0731 performanceDeepSeek V4 Flash benchmark TTFT TPOT throughput

06 / PRIMARY SOURCES

Source and qualification ledger

Official model facts, runtime support, and community measurements establish different claims. The ledger keeps those authorities visible.

01
DeepSeek · arXivarXiv:2606.1 · 2026-08-21
DeepSeek-V4 technical report
Flash architecture, MoE routing, hybrid attention, native context, preview benchmarks, and official efficiency comparisons.
02
DeepSeek · Hugging Face7872f01b1d1f · 2026-08-21
DeepSeek-V4-Flash-0731 model card
Current official checkpoint identity, mixed precision, DSpark serving, sampling, agent benchmarks, and license.
03
DeepSeek · Hugging Face7872f01b1d1f · 2026-08-21
DeepSeek-V4-Flash-0731 config.json
Released layer, expert, vocabulary, context, quantization, and DSpark configuration fields.
04
DeepSeek · Hugging Face7872f01b1d1f · 2026-08-21
DeepSeek-V4 encoding and DSML guide
Message roles, absence of a Jinja template, reasoning modes, DSML tool calls, and output parsing.
05
DeepSeek APIsha256:d9fc7 · 2026-08-20
Thinking mode and tool-call round trips
The reasoning-content continuity requirement for multi-step tool calls.
06
vLLM6f19519bb680 · 2026-08-21
DeepSeek-V4-Flash serving recipe
Checkpoint-specific minimum versions, supported hardware profiles, topology, parser flags, KV-cache choices, DSpark, and MTP boundaries.
07
vLLM1fe3a1571ac6 · 2026-08-21
Expert-parallel deployment
How TP, DP, and EP ranks compose for MoE layers and replicated or sharded attention.
08
vLLM1fe3a1571ac6 · 2026-08-21
Automatic prefix caching
Hash-based block reuse, workload requirements, and tenant-isolation salts.
09
vLLM1fe3a1571ac6 · 2026-08-21
Online serving benchmark CLI
Reproducible request distributions and TTFT, TPOT, ITL, latency, request, and token-throughput metrics.
10
vLLM1fe3a1571ac6 · 2026-08-21
Speculative decoding metrics
Drafted and accepted token counters needed to measure DSpark rather than assume a speedup.
11
vLLMccaabd8e0dba · 2026-08-21
DeepSeek V4 Flash B200 perf-eval workload
A directly inspectable 8×B200 TP2×DP4+EP benchmark topology for the preview checkpoint.
12
NVIDIAretrieved:20 · 2026-08-21
NVIDIA supported GPU identifiers
H100-SXM5 and H200-SXM5 identifiers, HBM capacities, the B200 GB100 identity, and RTX PRO 6000 Blackwell GB202 family identities.
13
NVIDIAretrieved:20 · 2026-08-21
NVIDIA GB300 NVL72 platform
The GB300 NVL72 rack identity, 72 Blackwell Ultra GPUs, 36 Grace CPUs, and aggregate GPU-memory and NVLink topology.
14
NVIDIAretrieved:20 · 2026-08-21
NVIDIA RTX PRO 6000 Blackwell family
The 96 GB GDDR7 ECC capacity and distinct Server, Workstation, and Max-Q Workstation editions that must be named in a result.
15
AMDretrieved:20 · 2026-08-21
AMD Ryzen AI Max+ 395 specifications
The Ryzen AI Max+ 395, Radeon 8060S, Strix Halo codename, and 128 GB maximum unified-memory identity.
16
SGLangretrieved:20 · 2026-08-21
DeepSeek V4 serving cookbook
Hardware-specific Flash serving support and topology boundaries for H100, H200, B200, GB300, and RTX PRO 6000 Blackwell.
17
pepuscz · GitHub957b9c2997a5 · 2026-08-21
Strix Halo DeepSeek V4 Flash deployment
A pinned single-host 128 GiB community qualification with Vulkan and ROCm paths, validation automation, and explicit portability limits.
18
pepuscz · GitHub957b9c2997a5 · 2026-08-21
Strix Halo benchmark protocol
Matched prompt-processing, generation, retrieval, cached-agent, and bounded quality measurements.
19
pepuscz · GitHubv1.1.0 · 2026-08-21
Strix Halo v1.1.0 qualification
The 524K Vulkan allocation and the measured 491,520-token cold-retrieval result on the specified host.

WORKLOAD-SPECIFIC EVIDENCE

Ask for the envelope your deployment actually needs.

Include checkpoint, target hardware, GPU or unified-memory count, runtime, concurrency, context and output distributions, latency and throughput objectives, reasoning effort, tool-call rate, precision, and whether local community weights are acceptable. Send only non-sensitive workload context.

“Serve DeepSeek-V4-Flash-0731 for 40 coding agents on 8 H200s, with 32K prompts, 8K outputs, Think High, DSML tool calls, and p95 TTFT below four seconds.”

Read the task llms.txt