Google Challenges NVIDIA with Open-Source TorchTPU AI Chip Software

SAN JOSE, California — Google has released TorchTPU, its native PyTorch software stack for Google Tensor Processing Units, as open source on GitHub, letting developers run existing PyTorch workloads on its artificial intelligence chips with minimal code changes.

Google's release of TorchTPU marks another step in its efforts to expand the use of its AI processors and strengthen its position in a market dominated by Nvidia. By making the software open source, Google aims to lower barriers for developers using PyTorch and encourage wider adoption of its Tensor Processing Units.

Nvidia has established a strong position in AI computing through CUDA, its software platform for accelerating computing workloads on GPUs. CUDA is widely supported by PyTorch and plays an important role in training and running AI models. This established software ecosystem has helped Nvidia maintain its competitive advantage, while Google's TorchTPU introduces another option for developers seeking to run PyTorch workloads on alternative AI hardware.

Demand for Nvidia's hardware also remains strong. This week, Chinese technology giants ByteDance, Alibaba and Tencent reportedly placed orders for more than 3 million Nvidia RTX PRO 5500 graphics cards, with a combined estimated value exceeding $35 billion.

Google's TPUs have traditionally relied heavily on JAX, a machine-learning framework widely used within the company, creating an additional learning curve for developers accustomed to PyTorch. TorchTPU aims to address this challenge by enabling developers to run PyTorch workloads directly on Google's AI processors, potentially reducing the effort required to adopt TPUs as an alternative to Nvidia GPUs.

TorchTPU was co-developed with the PyTorch team at Meta. Developers can debug models in eager mode, then move to production-level performance through torch.compile to optimize TPU utilization and reduce memory overhead, and the stack supports standard PyTorch APIs, advanced diagnostics and native distributed primitives out of the box.

Jana van Greunen, Director of PyTorch Engineering at Meta, said: “At Meta, advancing the frontier of AI requires immense compute scale and deep hardware diversity. We cannot be constrained by a single hardware architecture. TorchTPU is a massive step forward — allowing us to co-develop hardware flexibly with the core machine learning systems team while supporting major LLM, Ads, and Recommendation systems workloads on key production use cases. It enables us to deploy our PyTorch workloads to Google TPUs, provide direct feedback to the engineering teams, and build the ecosystem flexibility everyone needs.”

Google also partnered with Hugging Face so that popular open-weight models across the Hugging Face Hub run natively on TorchTPU. In Transformers v5, deploying models to TPUs is done by setting device_map to tpu, and the Diffusers library runs natively on TPUs with support for eager mode, torch.compile and tensor parallelism.

Jeff Boudier, VP Product of Hugging Face, said: “Hugging Face is committed to being an open, neutral platform where AI builders can choose the models, frameworks and hardware that work best for them. Hardware diversity is essential to making open-source AI accessible to everyone. Our collaboration with Google to bring native TorchTPU support to our core libraries, including Transformers, puts that commitment into practice. Together, we’re making it easier for the community to run popular models from the Hub efficiently on Google Cloud TPUs using familiar PyTorch workflows. Giving developers more choice in how and where they build helps the entire open-source ecosystem grow.”

Google has integrated TorchTPU into widely used open-source serving engines and paired them with Kubernetes-native fleet routing across TPUs and GPUs. A native PyTorch vLLM backend, developed with Inferact, supports continuous batching, quantization and multi-chip parallelism on TPU hardware without requiring changes to existing inference stacks.

Simon Mo, CEO and co-founder of Inferact, said: “Our mission at Inferact has always been to make high-performance inference accessible, reliable, and cost-effective across the entire AI ecosystem. We are thrilled to partner on the launch of TorchTPU and bring first-class support for vLLM to TPUs. By bridging PyTorch’s optimized TPU capabilities directly into vLLM, developers can immediately run production workloads on TPUs with the high throughput, low latency, and memory efficiency they expect from our stack.”

A native PyTorch backend for SGLang, developed with RadixArk, accelerates highly structured and agentic workloads on TPUs. SGLang uses RadixAttention to reuse KV cache across requests, so repeated prompt prefixes skip recomputation.

Banghua Zhu, CTO and co-founder of RadixArk, said: “At RadixArk, TorchTPU has supercharged our development velocity as we roll out SGLang support across diverse silicon. It’s given us a painless path to native TPU acceleration, passing massive efficiency gains and flexible compute choices directly down to our customers.”

Google is also releasing llm-d 1.0 with Red Hat, IBM and a community of more than 500 contributors and 25 organizations. On Google Kubernetes Engine (GKE), customers can run on either GPUs or Google TPUs with a common llm-d endpoint and choose the hardware that fits each workload.

Prof. Sashikumaar Ganesan, Founder of Zenteiq.ai, stated: “At Zenteiq, we use GKE Inference Gateway powered by llm-d to accelerate synthetic data generation for our sovereign Scientific Foundation Models. By leveraging load-aware and prefix-cache-aware routing, we have achieved over 4× higher aggregate inference throughput, increasing daily generated token capacity from 1.47 billion to 5.97 billion tokens per 1,000 TPU v6e accelerators. The open-source foundation of llm-d, integrated with GKE, enables us to efficiently serve multiple model families at scale without developing and maintaining a custom inference scheduler.”

TorchTitan, the PyTorch-native reference implementation for scalable distributed training, has a production-grade integration with TorchTPU and has been validated running with up to 8,000 Ironwood TPUs (TPU v7x), according to Google. TorchTPU’s training paths via TorchTitan demonstrated up to 52% Model Flops Utilization on Qwen 3 8B, with near-linear scaling across up to 8,192 Ironwood TPUs.

Google has upstreamed a TPU backend and an end-to-end reference recipe to the VeRL reinforcement learning framework. It has also added native TorchTPU support to Ray, so PyTorch data processing, training, reinforcement learning and serving can be orchestrated across CPUs, GPUs and TPUs in a single workflow.

Edward Oakes, Chief Architect of Anyscale, said: “At Ray Summit we talked about the PARK stack, PyTorch, AI models, Ray, and Kubernetes, because that’s what teams are building production AI on today. TorchTPU embeds Google TPUs directly in that industry-standard stack. We’ve also worked closely with the Google team to support TPUs as a first-class accelerator in Ray. Coupled with native PyTorch support, teams can now seamlessly develop, scale, and monitor their workloads on TPUs.”

LiteRT (formerly TensorFlow Lite), Google’s on-device AI inference, joins the PyTorch ecosystem. With LiteRT-Torch, a PyTorch model is converted with a single call and can run on Android, iOS, macOS, Windows, Linux, web and IoT devices.