HN
Today

DeepSeek V4 Flash on a Single AMD MI300X

This GitHub repository provides a detailed, production-ready configuration for deploying DeepSeek V4 Flash on a single AMD MI300X GPU. It meticulously addresses the unique challenges of AMD hardware, offering crucial patches, tuning tables, and a robust Docker Compose setup. The work demonstrates how to achieve impressive inference performance and capacity, making advanced LLMs accessible on cost-effective, alternative accelerators.

4
Score
0
Comments
#2
Highest Rank
11h
on Front Page
First Seen
Aug 4, 10:00 AM
Last Seen
Aug 4, 8:00 PM
Rank Over Time
72222344699

The Lowdown

This repository by ryanzhou offers a comprehensive solution for running the deepseek-ai/DeepSeek-V4-Flash-0731 large language model on a single AMD MI300X accelerator in a production environment. It provides all necessary configurations, patches, and tuning to overcome compatibility and performance hurdles not covered by official vLLM recipes, which often target NVIDIA or newer AMD hardware.

  • Core Components: The solution includes a Docker Compose stack, SHA-256-pinned file overlays, reference diffs against upstream projects, and specialized tuning tables.
  • Performance Metrics: On a single MI300X, it achieves a single-stream decode of 168.6 tok/s, prefill speeds of 7.9–8.5K tok/s, and supports 8 concurrent streams at 542 tok/s aggregate, with burst capacity up to 64 streams.
  • Hardware Rationale: The MI300X is chosen for its 192 GB of HBM3 memory, allowing the 304B-parameter model to fit entirely without quantization or offload, and its lower estimated cost compared to H100s.
  • Technical Challenges & Fixes: Key issues addressed include: differences in FP8 format (AMD's FNUZ vs. OCP standard), MoE routing bugs at high concurrency, causal speculative verification, CPU-KV synchronization, and untuned kernel shapes specific to the gfx942 architecture.
  • Repository Contributions: It adds correctness overlays for ROCm nightly, a validated serving configuration with probabilistic DSpark drafting, AITER GEMM tuning tables for gfx942 shapes, and a hybrid KV strategy using 20 GB GPU cache and 96 GiB CPU offload.
  • Deployment: The repository details prerequisites (MI300X, AMD driver, Docker Compose, sufficient RAM/disk), steps to pull images and models, prepare files, start the stack, and smoke-test the deployment.
  • Key Patches: Significant patches include a fix for MXFP4 routing bitmatrix padding and a critical FP8 format correction for the DeepSeek V4 Flash's Lightning Indexer cache, ensuring compatibility with MI300X's FNUZ FP8 implementation.
  • Optimizations: Performance gains come from tuning A8W8 GEMM shapes, fused SiLU, fast DeepSeek routing, BLOCK_H=64 sparse-prefill tiling, static K=7 for speculative decoding, a 2,048-token scheduler budget, and the hybrid GPU/CPU KV cache.
  • Production Notes: The guide advises careful HBM headroom management, clarifies the CPU KV tier's role, explains scheduler warnings, and emphasizes kernel warming and comprehensive correctness testing.

This work represents a significant effort to bridge the gap between cutting-edge LLMs and specific AMD hardware, providing a robust, optimized, and thoroughly documented solution for high-performance inference.