PyTorch-native TPU back end for vLLM
This project introduces vLLM-TorchTPU, a PyTorch-native backend that allows existing vLLM models to run efficiently on Google Cloud TPUs. It promises 'TPU speed' for your current PyTorch model definitions without requiring any code changes, addressing a significant pain point for developers seeking hardware-accelerated LLM inference. The tool integrates seamlessly with the vLLM ecosystem, offering familiar workflows and optimized performance.
The Lowdown
vLLM-TorchTPU is a new PyTorch-native backend designed for vLLM, enabling users to deploy and run large language models on Google Cloud TPUs. Packaged as vllm-torchtpu, this out-of-tree plugin aims to deliver the full vLLM experience—same CLI, OpenAI-compatible server, and model definitions—but at the accelerated speeds offered by TPUs, all without requiring any modifications to the model's PyTorch code.
Key aspects of vLLM-TorchTPU include:
- PyTorch-first Approach: Models are executed through vLLM's native PyTorch runner, eliminating the need for re-implementation in alternative frameworks.
- TPU-tuned Kernels: It incorporates a library of custom JAX/Pallas kernels specifically optimized for TPUs, covering critical operations like ragged Paged Attention, MoE (Mixture-of-Experts), and various quantized matrix multiplications (FP8/INT8/MXFP4).
- Built-in Scale-out: The system inherently supports various parallelism strategies including Tensor Parallelism (TP), Data Parallelism (DP), Pipeline Parallelism (PP), and Expert Parallelism (EP, for MoE models), allowing for scalable deployment across single and multi-host TPU slices.
- Standard vLLM Features: Users can leverage familiar vLLM functionalities such as speculative decoding, prefix caching, multi-LoRA support, structured output, and multimodal inputs, all integrated with the TPU backend.
- Hardware Support: The project officially recommends and validates on TPU v7x (Ironwood) with experimental support for older generations like TPU v6e/v5e/v5p.
- Ease of Use: Installation is straightforward via
pipor pre-built Docker containers, and serving models follows the standard vLLMservecommand and OpenAI-compatible API.
This initiative effectively bridges the gap for PyTorch-based LLM deployments, allowing developers to leverage the powerful acceleration of Google Cloud TPUs with minimal migration effort, thus streamlining the path to high-performance inference.