OpenAI Unveils Jalapeño AI Chip: Benchmark Results, Architecture, and the Inference Battle with Nvidia
At the Hot Chips engineering conference, OpenAI released the first verified benchmark data for its first-generation custom inference processor, Jalapeño. Developed in collaboration with Broadcom, the silicon demonstrates substantial gains in energy efficiency and token-generation latency over existing commercial systems, signaling a major shift in hyperscale AI infrastructure economics.
A Strategic Move into First-Party Silicon
As operating costs for large language models and autonomous agents escalate, leading software creators are turning to custom application-specific integrated circuits (ASICs) to control hardware spending and streamline compute capacity. OpenAI’s disclosure of Jalapeño marks its formal transition into hardware co-design, addressing the architectural bottlenecks of AI inference directly at the silicon level.
Richard Ho, Vice President of Hardware at OpenAI, emphasized during an engineering briefing that traditional commercial accelerators routinely force operators to compromise between overall token throughput and per-user response latency. Jalapeño was designed from inception to deliver both simultaneously, ensuring faster execution for complex, multi-step agentic workflows while maintaining lower power draw per rack.
InferenceX Benchmarks: Quantifying Speed and Energy Efficiency
To measure real-world performance across end-to-end request cycles, OpenAI benchmarked Jalapeño against leading commercial accelerators using the independent InferenceX suite developed by research firm SemiAnalysis. Testing spanned a spectrum of operating points, ranging from dense batch serving to highly interactive, low-latency agent queries.
Jalapeño consistently established a new Pareto frontier across internal and open-weight models, demonstrating significant performance advantages per kilowatt of power:
1. GPT-OSS 120B Benchmark
- Peak Throughput Efficiency: Delivered roughly 1.9x higher mixed tokens-per-second per kilowatt (85,448 mixed TPS/kW vs. 44,960 for an Nvidia GB200 system).
- End-to-End Latency: Reduced total request latency by approximately 1.7x (1.03 seconds vs. 1.80 seconds).
- Time Between Tokens (TBT): Achieved a minimum TBT of 0.69 ms compared to 1.87 ms on the reference system (translating to 1,459 vs. 535 tokens/second per user).
- Throughput at Prior Latency Baselines: Registered a 53.7x throughput increase when held to previous-generation generation speeds.
2. DeepSeek R1 (670B Parameters, MXFP4)
- Throughput Efficiency: Recorded 1.7x higher peak mixed throughput per kilowatt (19,641 vs. 11,781 for an Nvidia GB300 system).
- End-to-End Latency: Dropped latency by 3.6x (1.65 seconds vs. 5.99 seconds).
- Minimum TBT: Reached 1.43 ms vs. 5.90 ms, delivering 700 tokens/second per user compared to 169 tokens/second on competing hardware.
- Matched-TBT Scale: Achieved a 104.3x throughput advantage at matched baseline latency.
3. Kimi K2.5 (1-Trillion Parameter Scale)
- Peak Mixed TPS/kW: Outperformed reference GB300 hardware by approximately 1.5x (18,195 vs. 11,862).
- Latency Reduction: Lowered round-trip response time by 3.4x (1.56 seconds vs. 5.31 seconds).
- Token Generation Speed: Maintained a 1.44 ms minimum TBT compared to 5.48 ms on the comparison platform.
While the Jalapeño package carries a nominal thermal design power (TDP) rating of 700 watts, OpenAI engineers confirmed that sustained operational consumption remained at or below 550 watts across all validated production workloads.
Hardware-Software Co-Design: Eliminating the Memory Wall
Inference workloads encounter two distinct hardware bottlenecks throughout execution: the prefill phase (processing prompts and ingested context), which demands raw compute density, and the decode phase (generating subsequent tokens sequentially), which is heavily constrained by memory bandwidth. Moving state data across disparate chips or sockets introduces latency penalties that leave expensive compute cores sitting idle.
Jalapeño resolves these friction points through a unified, rack-scale architectural approach:
- Local KV Cache Retention: Model weights and Key-Value (KV) caches are explicitly mapped and maintained local to execution clusters, minimizing off-chip data transport.
- Integrated High-Domain Interconnects: A high-bandwidth fabric maintains entire model workloads within a single connected hardware domain, accelerating inter-core communication.
- Fungible Accelerator Pipeline: Hardware resources dynamically rebalance execution units between compute-heavy prefill operations and bandwidth-sensitive decode cycles without losing system state.
AI-Driven Chip Synthesis and Automated Programming
A distinctive aspect of the Jalapeño program is the extensive use of internal frontier models to automate both the physical design and software enablement of the silicon. OpenAI utilized earlier model generations to optimize arithmetic circuits and shorten verification loops, moving from initial architectural exploration to tapeout in nine months.
To program the chip, the team leveraged Codex alongside specialized models like GPT-Astra. This automated pipeline ported and optimized three complex open-weight model families (including DeepSeek R1 and Kimi K2.5) to production-level performance within two months. Internal benchmarks showed that AI-generated implementations of Mixture-of-Experts (MoE) routing and attention blocks executed 1.5x to 1.8x faster than code manually crafted by human kernel specialists.
Deployment Timeline and Multi-Generational Roadmap
OpenAI confirmed plans to commence initial internal deployments of Jalapeño in small volumes by late 2026, followed by broader infrastructure scale-out throughout 2027. The company reiterated that Jalapeño serves as the foundation of a multi-generational silicon roadmap: second-generation processors (Gen 2) are currently deep in development, while architecture planning for Gen 3 is actively underway.
Crucially, OpenAI clarified that it has no intention of selling or leasing Jalapeño chips to external enterprises or third-party cloud providers. Given OpenAI’s surging internal compute requirements for consumer services, developer APIs, and enterprise tiers, all custom silicon output is reserved strictly for first-party infrastructure.
Market Analysis: Implications for Nvidia and the ASIC Landscape
Industry analysts view OpenAI’s custom silicon as part of a broader structural transformation within the AI hardware supply chain. While Nvidia retains a commanding share of high-performance computing through its mature CUDA software platform, the rise of custom inference ASICs introduces competitive pressure on hardware profit margins.
Adrien Sanchez, Technology Analyst at Yole Group, noted that Jalapeño demonstrates how hyperscaler-designed silicon can effectively match or exceed Nvidia’s current architectures on inference efficiency-the segment of data center spending growing at the fastest rate. Alexander Harrowell, Senior Principal Analyst at Omdia, added that custom chips will significantly enhance unit economics by lowering power distribution and thermal dissipation expenses, projecting that custom ASICs will surpass standard GPUs in shipment volume by 2028.
Technical Nuance: HBM4 Memory Generations
Independent analysis from SemiAnalysis observed that part of Jalapeño’s benchmark edge stems from its integration of next-generation High Bandwidth Memory (HBM4), whereas the tested Blackwell systems rely on previous-generation memory configurations. From an architectural perspective, Jalapeño’s performance profile will more directly compete with Nvidia’s forthcoming Vera Rubin platform, which also incorporates HBM4. However, while Rubin hardware has already begun early sampling and delivery, Jalapeño is just transitioning from engineering verification to production qualification.
The Expanding Hyperscale Silicon Race
OpenAI joins an elite tier of technology giants developing proprietary accelerators to de-risk supply chain dependencies:
- Google: Continues long-term deployment of internal Tensor Processing Units (TPUs) for both training and inference.
- Meta: Securing multi-gigawatt custom ASIC capacity via long-term partnerships with Broadcom.
- Amazon Web Services (AWS): Expanding its Trainium and Inferentia footprints, backed by long-term capacity agreements with research labs like Anthropic.
- Hardware Startups: Innovative silicon architectures from Cerebras, SambaNova, D-Matrix, and Etched continue pushing throughput boundaries for specialized model architectures.
Despite the rollout of Jalapeño, OpenAI affirmed that partnerships with third-party semiconductor vendors remain critical. The company will continue to procure massive GPU clusters from Nvidia and accelerators from AMD and Cerebras to support frontier model pre-training. Ultimately, Jalapeño provides OpenAI with vital operational leverage-reducing the cost to serve while expanding the speed and reach of interactive artificial intelligence.
