Tokens per Watt Is the Inference P&L in One Number

The frame is an income statement: tokens are the revenue line of inference, watts are the cost line, and as hyperscalers move from building AI infrastructure to monetizing it, the ratio between the two decides who earns a return on the build-out. Power is the input that cannot be scaled on demand — I/O Fund cites an ERCOT queue of more than 438 GW in large-load interconnection requests, roughly 390 GW of it from data centers, against about 23 GW of capacity actually added across 2024–2025. Meanwhile each chip generation pulls more from the wall: roughly 700W for an H100, 1,200–1,400W for Blackwell-class parts, around 2,300W for Rubin. When new power is scarce and each accelerator burns more of it, the fastest revenue lever left is throughput inside the envelope already secured.

The binding constraint on that throughput, in their read, is memory rather than compute. Reasoning and agentic workloads inflate the KV cache — the working memory of inference — beyond what fits in HBM, forcing systems to either spill to slower memory tiers or recompute work already done; both leave expensive accelerators idle. The numbers behind the squeeze are the memory wall: over the twenty years to 2023, accelerator compute grew about 3× every two years while memory bandwidth grew 1.6× and capacity 2×. The gap persists into the current generation — Rubin's inference compute is cited at 5× Blackwell's, while its HBM bandwidth rises 2.8× and capacity just 1.5×. And because HBM comes packaged with the accelerator, buying more memory means buying more underutilized compute: HBM's share of accelerator bill of materials rises from roughly 52% in the B200 to an expected 62% in Rubin.

The proposed fix is architectural: offload engines that keep accelerators fed — Nvidia's proprietary CMX path on one side, an open ecosystem forming around CXL on the other. The claimed payoff is large. Nvidia's own figures have a Vera Rubin NVL72 rack paired with a Groq 3 LPX rack delivering up to 35× the token throughput per megawatt of a GB200 NVL72, and up to a 10× revenue opportunity versus Blackwell on trillion-parameter models. The cheaper levers are gone — hyperscaler PUEs of 1.09 to 1.17 leave little non-compute efficiency to harvest — and the cost asymmetry is stark: at cited prices, a terabyte of HBM acquired through B200 purchases runs beyond $200,000 against roughly $40,000 for server DDR5. Turning stranded memory into tokens is where the margin now lives.

The core idea When power is the scarce input and memory is the binding constraint, the operators who win are the ones who turn stranded accelerator-hours into tokens — scaling revenue inside a power envelope that no longer grows on demand.

That is why this reads as a monetization story rather than a hardware story. The build-out phase was judged on gigawatts secured and GPUs racked; the phase now starting gets judged on what each of those watts returns. A metric that divides revenue by the one input nobody can quickly buy more of has a way of becoming the number boards and analysts anchor on — and of repricing every layer of the stack that moves it.

Where it meets the Closelooknet frameworks

Tokens per watt is the monetization bridge between three frameworks Closelooknet already runs: the memory wall (compute outrunning memory), inference economics (tokens as the revenue unit) and the power constraint (the grid as the hard ceiling). This week put the whole chain on the tape: IBM's warning showed memory costs eating enterprise budgets from the demand side, ASML answered the capex question with a 30% EUV capacity add, and SK Hynix's HBM4 began shipping into the platform Nvidia named Vera Rubin — the build-out cycle the Rubin Build-Out 100 is constructed to track, layer by layer, from HBM memory to interconnects to AI factory systems. The offload-engine argument adds the missing lens: if memory cannot be scaled fast enough, the winners include whoever makes existing memory work harder. The concept now has a permanent home at Tokens per Watt in the 101, and our cut of I/O Fund's follow-up — the CMX-versus-CXL architecture race — is at Two Roads Out of the Memory Squeeze.

Closelooknet keeps this as a market-diary observation, not a recommendation — the signal is which metric the operators will be judged on, not a call on any single name.