OpenAI's Jalapeño Chip: The First Real Challenge to NVIDIA's Inference Dominance

At Hot Chips 2026 on August 25, OpenAI released the first benchmark results for Jalapeño, its custom inference chip built with Broadcom. The numbers are striking: 1.5 to 1.9 times more work per watt and 1.7 to 3.6 times lower latency than NVIDIA’s Blackwell GB200 and GB300, rising to 2.1 to 4.1 times on interactive workloads.

But the hardware isn’t the real story. The software is.

OpenAI built a new kernel programming language called Gluon that bypasses CUDA entirely — and used its own coding agent, Codex, to generate the kernels without human kernel engineering. The software stack went from zero to production-ready in under nine months after silicon returned from the fab.

This is the first full-stack challenge to NVIDIA’s inference dominance from a lab that generates enough AI traffic to actually test the theory at scale. And it landed on the same day NVIDIA reported quarterly earnings with a $91 billion revenue guide.

The Hardware: Jalapeño by the Numbers

Jalapeño is a reticle-sized ASIC co-developed with Broadcom on TSMC’s 3nm node:

• 13.4 PFLOPs of MXFP4 compute per die
• 15.4 TB/s of HBM4 bandwidth
• 700W TDP — lower than NVIDIA Rubin’s 900-1,150W
• 128 ASICs per two-rack system at 160 kW total
• Scalable to 2,048 chips across 16 racks

The benchmarks, run on SemiAnalysis’s InferenceX benchmark across three models (GPT-OSS, DeepSeek R1, and Kimi K2.5 1T), show impressive results:

• 1.5-1.9x more work per watt than NVIDIA GB200 and GB300
• 1.7-3.6x lower latency across the three models
• 2.1-4.1x on interactive workloads
• At concurrency-1 on Kimi K2.5: 700 tokens per second per user versus 100 tok/s for the next-best chip — a 7x gap

Richard Ho, OpenAI’s hardware head, put it plainly: “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly.”

The Software: Gluon and AI-Generated Kernels

The hardware story is interesting. The software story is more important.

OpenAI developed Gluon — a kernel programming language built on top of Triton — specifically to program Jalapeño. It preserves Triton’s SPMD programming model while exposing lower-level hardware abstractions that CUDA typically hides.

What makes this significant is how OpenAI bootstrapped the kernel library: Codex generated functional and efficient kernels without human kernel engineering, including MLA kernels for benchmarking DeepSeek. The chip’s software stack went from zero to production-ready in under nine months after silicon returned from the fab.

This is what actually threatens the CUDA moat. Custom silicon has existed for years from Google (TPU), Amazon (Trainium/Inferentia), and Apple (Neural Engine). What’s different is the mechanism: AI-generated kernels on a Triton-based stack make the usual “but you’d have to rebuild the entire software ecosystem” argument weaker.

SemiAnalysis noted that if Jalapeño succeeds, it signals that “the industry’s obsession over universal compilers is invalidated by frontier AI models.” That’s a significant claim. It’s also plausible.

The Caveats: What the Headlines Underplay

Several things the headlines are underplaying:

1. Blackwell, Not Rubin

The benchmarks compare Jalapeño against Blackwell (GB300), not NVIDIA’s Vera Rubin — which is already shipping to customers and uses the same HBM4 technology. SemiAnalysis itself called the Blackwell comparison “somewhat incomplete and unfair.” Against Rubin, the comparison on per-token cost and TCO is roughly equivalent, not a decisive Jalapeño win.

2. Multi-Token Prediction narrows the gap

Against a GB300 running multi-token prediction — the configuration NVIDIA deployments commonly use in production — the efficiency lead falls to roughly 1.5x. Still significant, but not the 4x that headlines imply.

3. Inference-Only

Jalapeño cannot train models at all. It’s an inference chip, not a training chip. NVIDIA remains dominant for training, and training is where the largest compute budgets are spent.

4. No Independent Verification

All InferenceX results were run inside OpenAI’s labs. SemiAnalysis verified the runs in person but did not execute the full suite independently. The benchmarks use single-turn 8k context — no AgentX results exist for multi-turn, long-context, or prefix-cache-heavy workloads that dominate actual production agentic deployments.

5. Deployment Timeline

Tiny volumes by end of 2026, broader production ramp in 2027, gigawatt-scale by H1 2028. If you’re calling OpenAI’s API today, nothing changes. Jalapeño is internal infrastructure — not for external sale. Self-hosted developers and NVIDIA customers aren’t directly affected.

The NVIDIA Earnings Context

The same week, NVIDIA reported fiscal Q2 2027 earnings after market close on August 26. The company guided to $91 billion in revenue (plus or minus 2%), representing roughly 95% year-over-year growth. Wall Street consensus sits slightly above at $92-93.5 billion.

But the headline number is beside the point. What moves the stock — and the mood of the entire AI trade — is the Q3 guide and any signal about China. NVIDIA’s guidance explicitly assumes zero data center compute revenue from China. Any signal that Chinese sales are resuming could move the stock more than the headline beat.

NVIDIA’s beat margin has been shrinking: 22.8% above guidance midpoint in fiscal 2024, down to roughly 5.5% in fiscal 2027’s first quarter. The stock has increasingly stopped rewarding beats on its own terms and started demanding forward proof that hyperscaler AI infrastructure spending isn’t slowing down.

The Jalapeño announcement, landing the same week, adds a new question for investors: will OpenAI — NVIDIA’s largest customer — start shifting inference workloads to its own chips? If OpenAI can run inference at 1.5-1.9x better efficiency on its own silicon, that’s NVIDIA revenue at risk.

What This Means for Business Leaders

1. Inference Costs Will Continue Falling

Jalapeño represents a new axis of competition in AI infrastructure: custom silicon optimized for inference. As OpenAI, Google, Amazon, and Meta all build custom chips, inference costs will continue falling. This means AI applications that were too expensive to run at scale last year may become economically viable this year.

The CTO Technology Advisory service helps organizations design AI infrastructure strategies that take advantage of falling inference costs — including model routing, caching, and the build-vs-buy decision for AI compute.

2. The CUDA Lock-In Is Weakening

For years, the argument against custom silicon was “you’d have to rebuild the entire software ecosystem.” OpenAI’s AI-generated kernels on a Triton-based stack make that argument weaker. If AI can generate production-quality kernels in months instead of years, the CUDA moat narrows.

This doesn’t mean CUDA is going away. NVIDIA’s software ecosystem remains the most mature and widely used. But it means the switching cost is lower than it was — and that changes the negotiating power of every large AI lab.

3. The Inference vs. Training Split Matters

Jalapeño is inference-only. NVIDIA remains dominant for training. If you’re building AI infrastructure, the decision is splitting into two: training infrastructure (NVIDIA, for now) and inference infrastructure (increasingly custom silicon). Organizations that conflate these two decisions will overpay for inference.

4. Production Workloads Need AgentX Benchmarks

The Jalapeño benchmarks use single-turn 8k context. But production agentic workloads — multi-turn, long-context, prefix-cache-heavy — are what enterprises actually run. No AgentX results exist for Jalapeño yet. Before making infrastructure decisions based on InferenceX benchmarks, wait for benchmarks that reflect real production workloads.

The AI Agent Automation Consulting service includes infrastructure assessment: evaluating your agent workloads against available inference options, designing model routing strategies, and building cost-optimized deployment architectures.

5. The AI Infrastructure Market Is Diversifying

The era of NVIDIA as the sole AI infrastructure provider is ending. OpenAI (Jalapeño), Google (TPU), Amazon (Trainium/Inferentia), Meta (MTIA), and Microsoft (Maia) are all building custom silicon. This diversification means more options, better pricing, and less vendor lock-in for organizations building AI systems.

The Bottom Line

OpenAI’s Jalapeño is not a NVIDIA killer. It’s an inference-only chip, compared against Blackwell rather than Rubin, with benchmarks run inside OpenAI’s labs. The deployment timeline stretches to 2028 for gigawatt-scale.

But Jalapeño is a CUDA moat challenger. The real story isn’t the 1.9x efficiency — it’s the AI-generated kernels on a Triton-based stack that went from zero to production in nine months. If AI can write the kernels that bypass CUDA, the software ecosystem argument that protected NVIDIA for a decade gets weaker.

NVIDIA’s Q2 results will show continued dominance. $91 billion in quarterly revenue is not a company in trouble. But the question for the next three years isn’t whether NVIDIA will keep growing — it’s whether the AI labs that buy NVIDIA’s chips will keep buying them for inference, or whether they’ll shift to their own silicon.

OpenAI just showed that the software moat is narrower than it looked. And they did it with the one resource NVIDIA can’t match: enough AI traffic to test, iterate, and deploy at scale.

The CUDA moat isn’t gone. But it’s being challenged — from the inside, by the largest customer, using AI to write the code that bypasses it. That’s a different kind of threat than NVIDIA has faced before.