Google's New TPUs Take Aim at Nvidia with Two-Chip Long-Context Design

Mountain View, California — Google has added two systems to its eighth generation of Tensor Processing Units (TPUs), the custom chips it builds for artificial intelligence: the TPU 8tx, aimed at confidential inference for regulated organizations, and the TPU 8i Max, designed for very large context windows and dense data center racks.

The company set out the two additions in a blog post by Amin Vahdat, Chief Technologist & SVP, AI & Infrastructure at Google. They follow the TPU 8t for large-scale training and the TPU 8i for real-time reasoning, which Google introduced at Google Cloud Next in April and said would be generally available later this year. Measured against the previous TPU generation, Google puts the 8t’s training gain at up to 2.7 times and the 8i’s improvement in inference price performance at 80%. Vahdat said: “This is the first time we have introduced four TPU systems in a single year.”

Google began designing TPUs more than a decade ago to meet its own needs in areas such as voice recognition and language translation. Over eight generations, the chips have come to train and serve the company’s Gemini models, run agentic workloads and power services for Google Cloud customers, and customers can now also install TPUs inside their own data centers. Google describes TPUs as a fast-growing business with nearly 100 customers, from frontier labs and trading firms to retailers and universities.

Vahdat said demand from outside Google is now shaping the hardware. In the post, he said: “While we’ve always co-designed our TPUs in partnership with Google DeepMind, we’re now increasingly getting feedback from external organizations on their specific needs. With computing moving from human to agent scale, businesses are asking for accelerators that can reason across AI models’ massive context windows. In addition, some highly regulated customers are asking for specialized encryption capabilities for unique workloads.”

Google draws a line between a new chip and a new system. Laying out billions of transistors on a silicon wafer takes years and is tied to long manufacturing cycles, the company said, while a system covers the compute, networking and storage assembled around the chip, including how they are packaged, and can be changed far faster for customers and researchers. Vahdat said: “When we built our very first TPU more than a decade ago, we knew AI software would evolve much faster than silicon manufacturing ever could.” Google therefore treated its TPUs as modular building blocks within its AI Hypercomputer, so engineers can rework packaging, high-speed memory, optical networking and security around them, which it says lets it deliver specialized supercomputers in months instead of years.

The first two eighth-generation chips set the baseline. The TPU 8t is meant for training very large foundation models with maximum goodput, which Google defines as the useful intelligence produced per watt of energy, while the TPU 8i handles reasoning and attacks the memory wall, reducing delays in communication between chips by as much as half. As companies move autonomous agents into production, Google said its two new configurations of those systems tackle two newer problems: data security at the hardware level, and context that runs to a million tokens.

The TPU 8tx, the third system in the generation, is tuned for secure, confidential inference and can also handle small-scale training. Google is pitching it at governments and regulated enterprises that run frontier or proprietary models and autonomous agents close to their most sensitive data. Data is usually encrypted while stored or moving across a network, but has to be decrypted in memory while it is processed. The 8tx, which uses the same silicon as the 8t and runs with 5th Generation AMD EPYC CPU hosts, moves protection into the hardware itself, so data and model weights stay protected and isolated even while in use.

Google also gave the 8tx Boardfly, the low-latency network it built for real-time reasoning, and said this cuts latency by up to 2x compared with the TPU 8t, with a near-zero performance penalty. Because one system combines training-grade compute with a fast inference network, teams can run and customize models in the same place: fine-tuning on private data, distilling large models into smaller ones and training agents through reinforcement learning, with no need to move sensitive data between separate clusters. That setup could make the 8tx relevant to banks and other regulated institutions weighing where to run AI on confidential records.

The fourth system, the TPU 8i Max, addresses the memory wall. When dozens of AI agents work at the same time across a whole software codebase or hours of video, Google said, memory bandwidth soon becomes the limiting factor, so it combined two low-latency TPU 8i chips on one package.

Compared with the TPU 8i, the 8i Max offers twice the compute, twice the SRAM and twice the HBM bandwidth, scaling compute and memory together so processors keep the data they need and large jobs do not stall. Since it fits double the silicon into the existing rack footprint, Google said data centers can install significantly more computing power per square foot and per watt.

The four systems give customers a dedicated accelerator for each stage of the agentic lifecycle while staying flexible across workloads, Google said: the 8t for scale-up training, the 8i for low-latency reasoning, the 8tx for confidential computing, and the 8i Max for the largest context windows and the densest racks.

Software is the other half of the push. When the first TPU arrived, only a handful of companies had workloads that needed one, and engineers relied on optimized frameworks and hand-tuned code. As AI adoption has spread, Google said it has worked to remove the barriers so that TPUs are as easy to use as general-purpose hardware. TorchTPU, which Google released as open source in October after co-developing it with Meta’s PyTorch team, now lets PyTorch code and Hugging Face models run natively on TPUs, with ready-made TPU backends for the vLLM and SGLang inference engines.

A new open-source TPU agent suite, now in preview, connects to coding tools including Claude Code, Cursor and Google Antigravity to handle TPU onboarding in minutes, find bottlenecks and autotune custom kernels from plain-language instructions. A new open-source XLA Serving Engine lets developers train models in Python and then serve them from a low-latency C++ runtime on TPUs, GPUs and CPUs. On the InferenceX and AgentX benchmarks, which measure realistic agentic coding and long-context tasks, Google said its native vLLM-TorchTPU stack posted top price-performance results, that new evaluations show the advantage holding across many frontier open models and serving stacks with high model accuracy, and that coming software optimizations on the same hardware will push those results further.

Two further tools target idle time and memory. AI agents keep state and can sit idle for seconds or even hours while they wait on a database lookup, an external tool or human feedback, so Google’s new Agent Compute Engine suspends idle sessions and resumes them automatically to reduce infrastructure costs. Google is also open-sourcing TPU Sync, a model-state layer for TPU clusters that joins on-chip HBM, host DRAM, shared storage and networking across the data center into one co-designed memory hierarchy. It moves large KV caches off the chip within milliseconds, streams context between separate prefill and decode nodes and keeps reinforcement learning weights in sync, reducing time to first token and KV transfer latency by more than 90%, according to Google.

Cloud Pathways, the Google DeepMind runtime behind Gemini, is now generally available on Google Kubernetes Engine (GKE) to every Google Cloud customer. With a single controller and the pathwaysutils library, users can run large TPU systems spread over several slices as though they were one local machine for pre-training, multihost inference and reinforcement learning, while colocated Python sidecars, slice hot-swapping in under a minute and elastic resharding keep long jobs going through hardware faults. GKE dynamic slicing, also now generally available, uses optical circuit switching (OCS) to separate how hardware is physically provisioned from how logical slices are formed, joining TPU cubes into larger topologies or splitting them into smaller isolated workloads on the fly. Google said this ends manual nodepool recreation and makes job startup up to 5x faster and fault recovery up to 4.5x faster.

Google said shipping four TPU systems and an open software stack in one year depends on compiler teams, networking engineers, silicon architects and Google DeepMind researchers working side by side, and it now uses AI to design the chips themselves, from DeepMind’s AlphaChip reinforcement learning system to Gemini producing chip layouts in hours rather than weeks. Vahdat said: “This is true co-design and a unique Google differentiator.” Beyond Google’s own cloud, Oracle Cloud Infrastructure is set to offer TPUs from calendar year 2027, under plans the two companies announced in October, according to Quartz.