OpenAI's Jalapeño: A Specialized Bet on the Inference-Heavy Future

The semiconductor landscape is shifting from the raw power of training to the efficiency of execution, and OpenAI is placing a massive bet on that transition. At the annual Hot Chips semiconductor development conference at Stanford, OpenAI provided its first detailed look at "Jalapeño," a custom AI accelerator developed in collaboration with Broadcom.

While Nvidia's Blackwell and the upcoming Rubin GPUs are designed as versatile tools for both training and inference, Jalapeño is built for a single, specific purpose: inference. This specialization is a strategic pivot. As first reported by The Register, OpenAI will likely continue to rely on Nvidia and AMD for the heavy lifting of model training due to the programmable nature of GPUs, but it intends to transition to its own silicon to optimize how those models are actually run.

**The Memory Bandwidth Play**

In the world of inference, compute is important, but memory bandwidth is the primary constraint. OpenAI's architectural choices reflect this. A single Jalapeño system consists of 128 accelerators, delivering 1.7 exaFLOPS of 4-bit compute and 27.5 TB of HBM4.

When compared to Nvidia’s GB200 NVL72 and GB300 NVL72 rack systems—as well as AMD’s Helios—the trade-offs become clear. The Register reports that while the AMD and Nvidia racks are faster in terms of raw compute (delivering 1.46x to 2x more) and offer up to 12 percent more memory, OpenAI's rack outperforms them in the metric that matters most for inference: memory bandwidth. Specifically, the competing racks provide only 85 percent of the memory bandwidth found in the Jalapeño system.

**Benchmarking the "Inference Beast"**

OpenAI shared early benchmarks using SemiAnalysis’ InferenceX suite, testing against models including Kimi K2.5, DeepSeek R1, and GPT-OSS-120B. The results suggest a significant leap in efficiency over the competition:

* **Peak Throughput:** Jalapeño-based systems delivered between 1.5x and 1.9x more "AI work." * **End-to-End Latency:** The systems showed a 1.7x to 3.6x reduction in latency. * **Ultra-Low-Latency:** In this specific segment, OpenAI claims its chips are 2.1x to 4.1x faster.

It is important to note, as The Register points out, that these comparisons excluded speculative decoding, and the InferenceX tests are considered unofficial.

**Architectural Efficiency and the SRAM Strategy**

To achieve these gains, OpenAI is focusing on minimizing data movement. Richard Ho, VP of hardware at OpenAI, explained that the chip is designed to keep intermediate operations and key-value (KV) caches—which track state—local to the chip. By explicitly placing model state locally, the system can activate the necessary compute, memory, and networking for each inference phase without the delays associated with moving data.

This approach likely involves a large SRAM cache, a design choice seen in AI XPU designs from companies such as dMatrix and SambaNova. Unlike some specialized racks, such as Nvidia's Groq LPX, Jalapeño is designed to handle both the compute-heavy prefill operations (processing prompts) and the memory-bandwidth-intensive decode phase (generating tokens).

**The Roadmap to Volume**

Jalapeño represents the first in a series of custom silicon designed (in part) by AI, for AI. The hardware is expected to begin trickling out later this year, with volume production slated for 2027. This timeline puts it in direct competition with the ramp-up of Nvidia’s Rubin and AMD’s MI455X GPUs, which are also expected in early 2027.

From a power perspective, while OpenAI has not released official system-level consumption figures, The Register estimates that each rack will likely use between 40 and 60 percent of the power required by competing GPU systems. Each individual accelerator is capable of 13.4 petaFLOPS at MXFP4, supported by 216 GB of HBM4 memory providing 15.4 TB/s of bandwidth.