A production deployment configuration for running DeepSeek-V4-Flash-0731 on a single AMD MI300X GPU. It bundles Docker Compose orchestration, performance patches, kernel tuning tables and reproducibility documentation. The model itself was released in late July under the MIT licence; this is the serving stack around it.

What are the reported throughput numbers?

Single-stream decode reaches 168.6 tokens per second. Tuned prefill runs at 7,900 to 8,500 tokens per second. Eight concurrent streams aggregate to 542 tokens per second, and a 64-stream burst reaches 830 tokens per second without running out of memory. These figures are author-reported from the repository and have not been independently replicated.

Why does a single GPU work for a 304B model?

The MI300X carries 192GB of HBM3 at 5.3 TB/s, roughly 2.4 times the memory of an H100. DeepSeek V4 Flash weighs about 156GB unquantized, so the full model fits in one card's memory with room for a KV cache. That removes tensor parallelism, the interconnect between GPUs, and the quantization step from the deployment entirely.

What is the FNUZ FP8 problem?

AMD hardware uses the FNUZ variant of 8-bit floating point, which differs from the OCP standard format used on NVIDIA hardware by up to a factor of two in scale. Weights or scaling factors quantized under one convention and loaded under the other produce silently wrong numerics rather than an error. The repository ships fixes for this, and it is the kind of bug that costs weeks if you meet it without warning.

What does the stack depend on?

A digest-pinned vLLM ROCm nightly build, specifically 0.26.1rc1.dev229+g124154a88.rocm723, with probabilistic speculative decoding and a hybrid KV cache split across 20GB of GPU memory and 96GB of CPU memory. Hardware requirements beyond the GPU include roughly 235GB of system RAM for the CPU cache tier and about 500GB of disk.

Is the context window really 1 million tokens?

The architecture supports it; the repository validates 256K. Those are different claims and the distinction matters for capacity planning. Treat 256K as the supported figure for this deployment and anything beyond it as untested on this configuration.

Should a team switch to AMD based on this?

Not on this alone. What this demonstrates is that the path exists and has been walked, which was genuinely uncertain for frontier-scale mixture-of-experts serving until recently. But the stack depends on a pinned nightly build and hand-tuned kernel tables, which is a maintenance commitment rather than a one-time migration. It is a strong argument for running an evaluation, and a weak argument for moving production this quarter.

A 304B Model at 168 Tokens a Second on One AMD GPU

DeepSeek-V4-Flash-0731 is 304 billion parameters, MIT-licensed, and about 156GB on disk unquantized. A deployment configuration published this week runs it on one AMD MI300X at 168.6 tokens per second single-stream, with tuned prefill between 7,900 and 8,500 tokens per second. It reached 370 points on Hacker News.

The throughput number is the headline and it is the least interesting thing in the repository.

What is interesting is the list of things that had to be fixed first.

Executive summary

DeepSeek-V4-Flash-0731: 304B parameters, MIT licence, 256K context validated (architecture supports 1M), ~156GB unquantized.
Serves on one MI300X — 192GB HBM3, 5.3 TB/s, roughly 2.4× an H100’s memory at an estimated ~50% of list price.
168.6 tok/s single-stream decode; 7.9–8.5K tok/s tuned prefill; 542 tok/s across 8 streams; 830 tok/s on a 64-stream burst with no OOM.
Required fixes for AMD’s FNUZ FP8 format, which diverges from the OCP standard by up to 2× in scale.
Also required MoE routing corrections at high concurrency and custom GEMM tuning worth a 42–62% decode speedup.
Depends on a digest-pinned vLLM ROCm nightly, ~235GB system RAM for a CPU KV tier, and ~500GB disk.
All throughput figures are author-reported and unreplicated.

The arithmetic that 192GB changes

Serving a 304B model has, until now, meant one of two compromises.

Quantize it, and accept an accuracy cost that is hard to characterise and harder to explain to whoever owns the product metric. Or split it across GPUs, and accept tensor parallelism — which means an interconnect in the critical path of every token, a topology that has to be right, and a class of performance bug that only appears under load.

192GB of HBM3 removes both. A 156GB model fits with room left for a KV cache. There is no quantization step, so there is no accuracy question to answer. There is no second GPU, so there is no interconnect, no NCCL/RCCL tuning, no all-reduce in the decode loop, and no partial-failure mode where one card in the group degrades.

That simplification is worth more operationally than the tokens per second. A single-process, single-device deployment is a fundamentally easier thing to run, monitor and debug than a distributed one, and the difference compounds over a year of on-call.

The cost comparison follows from there. The repository estimates the MI300X at roughly half an H100’s list price. Whatever the real number is for a given buyer, the shape holds: one card instead of several, at a lower price per card, serving a model that is free to use.

The patches are the actual content

Strip out the throughput table and what remains is a list of things that were broken, which is far more useful to anyone considering this path.

FNUZ FP8. AMD’s 8-bit float format is not the OCP standard format. The scale differs by up to a factor of two. This is the worst category of bug: quantized weights or scaling factors produced under one convention and consumed under the other do not raise an error — they produce plausible-looking output that is wrong. You find it by noticing your evaluation scores are mysteriously soft, which can take a very long time if you were not expecting a numerics problem. Anyone porting quantized inference to AMD hardware will meet this. Knowing it exists in advance is worth more than the rest of the repository.

MoE routing under concurrency. Mixture-of-experts routing is where MoE serving implementations tend to break, and it breaks at concurrency rather than in single-stream tests — which is exactly the regime that benchmarks under-cover and production over-samples. That the repository specifically calls out corrections here, and reports a 64-stream burst without OOM, suggests the failure was found the hard way.

GEMM tuning worth 42–62%. Custom kernel tuning tables producing a half-again decode speedup is a large number, and it is also a warning. It means the out-of-box kernel selection on this hardware was leaving roughly a third of the achievable performance unclaimed. Anyone benchmarking ROCm without tuning is measuring the wrong thing and will conclude, incorrectly, that the hardware is slower than it is.

Causal speculative verification for DSpark. Speculative decoding needs its verification step to preserve the model’s output distribution exactly, or you have quietly traded correctness for latency. This is the component most worth auditing yourself before trusting the throughput figures, because a speculative decoder that verifies loosely is fast for uninteresting reasons.

What “one GPU” quietly excludes

The headline says single GPU. The requirements list says roughly 235GB of system RAM and about 500GB of disk, because the KV cache is hybrid: 20GB on the GPU and 96GB on the CPU.

That is a defensible design — a CPU cache tier is how you get long context and high concurrency out of a fixed HBM budget, and it is the same idea appearing across the inference stack this month. But it means the deployment target is a substantial server, not a workstation with a good GPU in it, and anyone sizing from the headline will size wrong.

The other dependency deserves more attention than it usually gets. The stack pins a vLLM ROCm nightly — 0.26.1rc1.dev229+g124154a88.rocm723. Digest-pinning is exactly right for reproducibility and it is the correct engineering decision here. It is also an accurate description of the maturity level: this configuration is validated against one specific unreleased build, and the upgrade path to the next one is unvalidated by construction. Combined with hand-tuned kernel tables, the honest characterisation is that this is a maintained artefact, not a stable platform.

The claim to be careful with

The model card says 1M context in the architecture. The repository validates 256K.

Those are different statements and the gap between them is where capacity planning goes wrong. Architectural support means the position encoding and attention mechanism do not forbid it. Validation means somebody ran it and checked the output was still coherent and the memory did not blow up. Only the second one is a number you can plan against.

Use 256K. If you need more, the burden of testing is yours.

Likewise, every throughput figure here is author-reported. The repository is unusually good about this — it publishes tuning tables and reproduction documentation, which is more than most performance claims come with, and it makes independent replication genuinely possible rather than nominally possible. But possible is not done. Until someone unaffiliated posts numbers, treat the table as a well-documented claim.

Should you move

No, not on the strength of one repository. But the question changed shape this week, and that is worth registering.

The reason AMD has not been a serious option for frontier-scale serving was never the silicon. The MI300X’s memory and bandwidth have been favourable for a while. It was that nobody wanted to be the team that discovered the FNUZ divergence, the routing bug and the kernel-tuning gap on their own schedule, with production traffic waiting.

Someone has now been that team and published the results. That does not make the path smooth — a pinned nightly and hand-tuned GEMM tables are not a smooth path — but it converts an unknown-cost migration into a known-cost one, and unknown cost is what actually blocks evaluations.

The reasonable move is an evaluation, budgeted realistically. Not a quarter-scale migration on the basis of a number in a README.

The verdict

A 304B MIT-licensed model serving at 168 tokens a second on a single card that costs roughly half an H100 is a genuine change in the cost floor for self-hosted frontier inference. If the numbers hold up under replication, the economics are hard to argue with.

But read the patch list before the benchmark table. The benchmark table tells you what one team achieved on tuned hardware with a pinned build. The patch list tells you what it will cost you to get there — and that is the number that determines whether this is a good idea for your team specifically.