Episode 7 — Wafer-Scale Computing After Cerebras: A Deep Dive on the Silicon Tradeoffs Behind AI Hardware

Introducing Raminda Madurawe and today's topic

Sunil Baliga: Friday afternoon, Token Drop time — this week we have a special guest, Raminda Madurawe, founder and CEO of AxPro Semi. I've known Raminda a long time; he's one of the sharpest technical minds I know in the chip world. Today we're talking about the broader trends kicked off by Cerebras, and I couldn't think of a better person to unpack that with. As always, we're keeping this at the level of general trends — not talking about any specific company's products.

Cerebras just had its IPO — and interestingly, their CMO, Julie Choi, is someone Sajjad and I worked with years ago at WIPRO when she was still at Intel. She later moved to MosaicML, and after Databricks acquired MosaicML, she ended up as Cerebras's chief marketing officer. Congratulations to her on a genuinely strong IPO.

What I want to dig into: does this herald something bigger for the AI industry? Cerebras has its own chips, you can buy their systems, or use their cloud. Are we going to see more of this — companies building specialized chips, and other companies just calling into them via an API without worrying about the chip layer at all? The way we might call Claude or OpenAI via an API today without caring whether it's running on AWS or Azure underneath — is that the direction hardware is heading too, with cloud providers increasingly running workloads on chips like Cerebras behind the scenes? Raminda, I'll open it up to you.

The "silicon tax": three costs of running someone else's API on your hardware

Raminda Madurawe: Thanks for having me. Your point is well taken. If you have an API that can run on platform X, Y, or Z, the person calling that API generally doesn't care which platform it runs on — they'll find whatever's most optimal on cost, power, or whatever metric matters to them, and run it there.

But on the hardware side, the real question we've all wrestled with is: can you run any API transparently on any hardware, or do you have to pay a tax to run someone else's API on your silicon? The answer is: you always pay a tax. The question is what form that tax takes.

First, there's a software tax — you may technically be able to run someone's API, but you often have to build additional software just to translate it onto your hardware. Second, there's an indirection tax — your silicon may not be especially efficient at running that specific API, so you run it anyway, just less efficiently, with real overhead. Third, there's what I'd call an accuracy, utilization, or efficiency tax — you may not be able to deliver the same precision or performance the original API expected. Those are the three variables everyone in this space is constantly trying to optimize.

Why AI needed accelerators in the first place

Sunil: Let's stay general rather than Cerebras-specific.

Raminda: Sure — let's talk architecture. If you go back to classic CPU architecture, it was never really built to handle AI workloads well — that's exactly why we've seen this wave of AI accelerators emerge. Once you accept that you need some kind of dedicated AI accelerator architecture, an interesting counterfactual comes up: if a CPU could have handled AI workloads adequately, would anyone have bothered building a dedicated accelerator at all? Nobody really knows — but the honest answer is probably not; nobody would have gone to the trouble. But here we are, and accelerators are now a given.

Everyone essentially converged on GPU-style architecture, largely because it already existed — GPUs were originally built for a completely different purpose, but they happened to be available and handled what CPUs couldn't, making them a convenient foundation to piggyback AI workloads onto.

That naturally raises the question: is there a fundamentally different architecture that would do the job better? The answer comes down to a mathematical question — what data graph do you actually need to compute for your target mathematical function? GPUs implement one particular kind of data graph. From there, you can ask whether a more efficient data graph is possible — faster, lower power, whatever metric matters — and a lot of real engineering thought has gone into exploring that path.

Two competing philosophies: instruction sets vs. massive fixed silicon

Raminda: There's a parallel question, though: if you want computing at its most efficient possible operating point, what does that hardware actually look like? The answer is a custom ASIC built to do exactly the one function you need — nothing is more efficient than purpose-built silicon for a single task. But that comes with a flexibility tax: if you need to do twenty different things, do you build twenty different ASICs? That becomes its own overhead problem — you can't reasonably pay for twenty or thirty different custom chips.

So the industry has moved toward domain-specific accelerators, betting that a broad enough class of applications will use the same piece of silicon, amortizing its cost and making it appealing to a wider group of users. On one side, you have GPUs and NPUs offering a smaller instruction set — a defined set of functions you access through instructions to get better accelerated computing. On the other side, a different group of engineers said: instead, we'll build you an enormous, fixed data graph, requiring a large footprint of silicon, that lets you process a large unified graph in parallel and get much higher throughput simply by throwing more silicon at the problem. Both approaches have been pursued seriously — you've seen companies building large-footprint silicon solutions, alongside companies building this other, more traditional instruction-based accelerated computing.

Cloud economics: sharing silicon across many users

Raminda: In the cloud specifically, the question shifts again, because you have so many simultaneous users: can I share this piece of silicon across multiple users efficiently? How do I divide time on it? And if more users happen to be running the exact same underlying model, I don't need to reload a completely different set of model parameters for each one — I can just swap in user-specific content on top of a shared base, which is a real efficiency advantage.

Where does this all end? Cloud vs. local, and marketing vs. reality

Raminda: You asked where this ends. My honest view is that the cloud is genuinely excellent for one category of task: understanding what everyone else is doing, summarizing broad information — I do this constantly, and I'm sure you all do too. Sometimes that's trivial, like finding a restaurant. But for real work — and my company uses AI very heavily, I'm sure yours does too — I genuinely don't want my proprietary information going out into a shared cloud for some external model to learn from. I want that applied against my own private data, and today that's genuinely restricted, so for a lot of real work, we end up running things locally instead.

I keep coming back to the idea: wouldn't it be nice if every unit of compute could effectively be its own small data center? Wouldn't you rather have your own work processed locally, at your own pace, without a shared model that's simultaneously factoring in what someone in London is doing — something you don't actually care about? Wouldn't you rather have something more custom to you specifically, more accurate, with less hallucination? My sense is this world is going to split into these two camps, and we're already seeing that play out — plenty of AI hardware entering the market on both sides.

But if you actually read the specs closely versus the marketing, there's a real gap between the two. In the cloud, essentially every major model — OpenAI or otherwise — runs in FP16 precision, arguably because dropping to a lower precision would cost real accuracy or introduce a level of uncertainty nobody's willing to accept; otherwise everyone would already be using more aggressive quantization, since it's meaningfully cheaper on power. But an "AI PC" will happily advertise that 4-bit quantization is "good enough." If 4-bit is genuinely good enough on a PC, why wasn't it good enough in the cloud? I honestly don't know the answer — my suspicion is there's real marketing spin at play: either FP4 truly is fine and everyone's overcautious in the cloud, or you're being quietly told to accept lower accuracy because you're on a cheaper solution, and if you want better accuracy, pay more and go to the cloud.

Can you partition a wafer-scale chip to serve multiple models at once?

Sajjad Khazipura: Thanks for setting the stage. The tradeoffs and design choices around why you'd run something on a PC versus the cloud, or vice versa, matter a lot — and there's also the service-provider angle: as a provider, I might want to support the widest possible diversity of models, some very large, some small, and serve all of them efficiently. That's a problem we've genuinely struggled with. The typical choice comes down to hosting and serving the most popular models directly, then offering APIs for a long tail of less common models that customers can spin up on-device as needed, rather than running everything continuously yourself.

Given the wafer-scale computing paradigm we're discussing, and the fact that transistor counts on these chips are only going to keep increasing — is anyone actually talking about partitioning a large wafer-scale estate to run a genuine diversity of models simultaneously, some large and some small, on the same physical wafer, serving multiple different applications at once?

Raminda: The short answer is: it should be possible. A wafer-scale integrated design typically has some kind of control layer capable of running control functions across a defined region of the chip, since the wafer is inherently divided by physical space. In principle, you could partition it into quadrants — but that requires the hardware to be architected upfront to feed data that way. Even at wafer scale, a trillion-parameter model will never fit entirely on-chip. You break it down layer by layer, and even a single layer can sometimes exceed available capacity, which raises the real question: what if you only load a quarter of a layer, and run four of those in parallel? That has to be an explicit hardware design decision made in advance, because data has to be fed into the wafer through some kind of bus architecture, and that bus needs to know how to distribute data into each quadrant. If that mechanism doesn't exist by design, you simply can't do it after the fact. Many of these wafer-scale architectures use a flat memory architecture instead — meaning they don't necessarily partition memory into quadrants for loading; they load systematically across memory blocks and process through it that way.

Sajjad: In Cerebras's case specifically, they use a network of small compute cells — each cell has some compute, a router, and some local memory. One thing that intrigued me: even with such an enormous die, the total on-chip memory is still only in the tens of gigabytes — not dramatically different from a high-end NVIDIA GPU, even though NVIDIA relies on HBM (off-die, but tightly connected) rather than SRAM directly on the die.

Sam Pooni: It's roughly 24GB on-chip, I believe.

Raminda: From an SRAM perspective, that's actually enormous — but compare it to a single DRAM chip, which can hold 32GB on its own, and it doesn't sound as impressive.

Sajjad: Right — that's exactly the SRAM-versus-DRAM distinction worth unpacking.

Why SRAM doesn't scale the way DRAM does

Raminda: That's a question I asked myself two or three years ago while building my own company. It comes down to a few things. First, DRAM cells are inherently smaller than SRAM cells — but that's not even the primary factor. The bigger factor is that DRAM manufacturers have gotten extremely good at building vertically — stacking many layers within a small footprint. A roughly 190 square millimeter DRAM chip can deliver around 32GB. SRAM's per-bit cell is, to start with, four or five times larger, and there's no equivalent technology today for stacking SRAM layers on top of each other — it has to occupy flat real estate, and real estate is expensive.

On top of that, SRAM essentially hasn't scaled with newer process nodes the way logic has. Going from 7nm to 5nm to 3nm barely shrinks the SRAM cell — you get real power and performance gains on logic at smaller nodes, but almost nothing on SRAM density.

So the real design question becomes: do you want SRAM spread out as real estate, with data physically moving from block to block since every block has its own local SRAM? Or do you centralize around fewer processing "houses," bringing data to the house, doing the operation, then bringing in the next set of parameters to that same house? The second approach needs less real estate but more power, and can be less efficient overall because you're waiting for data to arrive each time rather than having it already local.

Memory bandwidth, not compute, is the real bottleneck

Sam: Building on that — when it comes to something like tensor processing units, data gets shunted to a specialized processing unit, which involves real data movement. What I'm getting at is: the core premise behind Cerebras's approach seems to be that memory bandwidth, not compute, is the actual bottleneck in AI workloads — so they put all the memory directly on the same piece of silicon as the compute, and make that piece of silicon as physically large as possible. Is that a fair read?

Raminda: Yes — and thank you for raising that, because there are really two separate things worth untangling: what exactly do we mean by memory bandwidth, versus compute bandwidth?

A CPU typically pulls data from cache memory, with separate instruction and data caches. The data cache deals with operands — small, finite chunks of bytes, never SIMD-style batches, because CPUs are built for general-purpose work with a narrow pipeline. So you're inherently bandwidth-limited on the data side, because the CPU couldn't use more even if it had it.

GPUs have more flexibility, since they support SIMD instructions and can be architected with a wider bus — a single instruction might move 128 bytes or more, all in parallel, giving genuinely higher bandwidth. But even GPUs eventually run into memory bandwidth limits, because that data ultimately has to come from external memory over a separate, external bus rather than the chip's internal bus. That's exactly why HBM exists — pushing bus width from something like a 64-wire DDR-style bus up toward 2,048 pins in upcoming HBM4-generation designs.

CPUs also face a distribution challenge — feeding data to a CPU core and an NPU core simultaneously means feeding disparate compute types over the same wire mesh, which GPUs don't have to deal with since they're architecturally uniform, though GPUs have their own bandwidth ceiling as a result.

Wafer-scale designs like Cerebras and SambaNova have a real structural advantage here: because so much memory sits directly on-die, once a die (or in Cerebras's case, the equivalent of many dies across a whole wafer) is loaded, that data can be processed extremely quickly, since it's genuinely on-chip rather than reached over an external bus — effectively dozens of chips' worth of parallel processing. But they still face the same fundamental problem once that batch of data is exhausted: the next set of data still has to come from somewhere outside. It becomes a divide-and-conquer problem, and you'll win in some scenarios and lose in others — you can't fully win on both memory locality and unlimited capacity simultaneously.

Sam: The memory bandwidth number I've seen cited is around 21 petabytes per second for Cerebras. And there's another interesting point buried in the model side: even a very large model — something like a 253B-parameter model — has to be sharded across GPUs, typically in a cluster of 8 GPUs (say, H100s) connected via NVLink, requiring operations like all-reduce across those GPUs. With Cerebras's wafer-scale approach, that inter-chip networking largely goes away for a single model — that's one of their central claims, and it may also sidestep some of the yield challenges that come with stitching together many discrete chips.

NVLink, latency, and the case for wafer-scale

Raminda: NVLink is a genuinely nice piece of engineering — quick bit of history: PCIe, which most of us use for interconnect, carries a real penalty every time you want to get on the shared bus, since it's shared across many connected devices, adding several microseconds of latency before communication can even start. NVLink is more of a point-to-point connection, giving meaningfully better latency — something like 4 microseconds versus 10 for PCIe — though it still relies on a roughly 64-wire bus with PAM4-style data compression to move data between domains.

In some recent high-performance "superchip" designs, a CPU-to-GPU link over NVLink achieves something on the order of 900 gigabytes per second between, say, a 72-core CPU and a large GPU cluster — and that same connectivity extends GPU-to-GPU as well. But every time you communicate across that link, or access virtual memory across it, you're paying that link's latency; all CPU calls have to traverse it. A wafer-scale engine, where everything sits physically adjacent on the same wafer, doesn't have this link at all — so I'd expect a meaningful performance and possibly power advantage there. It's a genuinely hard structural problem for any GPU-CPU architecture built from separate chips; you could theoretically place them side-by-side on the same package, though GPU efficiency might suffer if the CPU and GPU aren't as tightly integrated. A more elegant long-term solution might be finding a way to integrate that entire interaction inside a single unified core, eliminating the latency and bandwidth penalty altogether and meaningfully improving interoperability — that's a real future possibility, though not something available today.

The cost of "throwing silicon" at the problem

Raminda: Here's the part I've always been a little uneasy about, Sunil — when we were designing chips early in our careers, the entire goal was making the smallest possible piece of silicon. Now the goal has essentially inverted: how many of these can we build, how big can we make it? Suddenly this isn't a local solution anymore — it's inherently centralized, requiring facilities that can physically host and power these massive systems. What's the power draw, Sajjad — something like 20 kilowatts for a single wafer? Some of these GPU racks run around 120 kilowatts.

Sunil: Wait, kilowatts? I thought you meant watts. That's enormous.

Sam: I've seen figures closer to 500 kilowatts for some configurations — reportedly fitting in about a third of a standard data center rack.

Raminda: Right — everything's liquid-cooled now, with elaborate cooling fin designs and a whole new category of cooling companies. I told one cooling vendor recently: the best cooling solution is not needing cooling in the first place — everything else is really just solving a problem that arguably shouldn't have been created to begin with.

Local memory, cache refills, and the "two-player game" of compute vs. memory

Sam: Since you raised HBM bandwidth — roughly 3 terabytes per second on an H100, around 8 terabytes per second on a B200, which is essentially a ceiling — on Cerebras, the model weights sit in SRAM literally nanometers away from the compute itself, giving them that roughly 21 petabytes-per-second, single-cycle access figure. Is that essentially the entire reason Cerebras dramatically outpaces GPUs on raw token generation speed?

Raminda: Yes — and that's a great point, because there's real confusion out there about what "bandwidth" actually captures. When you operate purely out of cache, what's the total achievable bandwidth in tera-operations? Even a 72-core CPU running at 4GHz will post an impressively large number here. This is why an NPU with 6 or 8 cores — something like Apple's M4, in roughly that ballpark — can hit around 48 teraflops, but only sustain that for roughly 100 cycles before the cache flushes and has to be refilled.

Now imagine expanding that from 8 cores (roughly 2–4 square millimeters) across an entire wafer — something like 40,000 square millimeters, the scale of a Cerebras-class solution. Scale that math up, and yes, you get an enormous petaflop figure. But if you scaled an equivalent NPU design to that same physical area, you'd likely see an even larger number, potentially an order of magnitude higher — because the real constraint isn't peak throughput, it's how long it takes to refill local memory once it's exhausted. It's genuinely a two-variable game, not a single number.

Von Neumann vs. dataflow architectures: Cerebras, SambaNova, and Groq

Sajjad: That raises another architectural question. We're still fundamentally discussing classical von Neumann architectures here. What about true dataflow architectures instead — a wafer-scale engine with local SRAM and local compute everywhere, fully interconnected, where model weights and code are distributed across the wafer, and inference data simply flows through it — processed and forwarded at each unit in a continuous pipeline, closer to genuine dataflow architecture? SambaNova, I believe, attempted something like this — I'm not sure where they've landed. What's your read on Cerebras versus SambaNova versus a truer dataflow definition?

Raminda: Glad you picked those two specifically — I've actually spent real time looking at both.

Sam: And while you're at it, I'd love your take on Groq's LPU compared to SambaNova's reconfigurable dataflow units.

Raminda: On the Cerebras side: they still run a custom instruction-processing unit with a fairly small instruction set, looping and executing within that instruction capability — but they do offer a genuinely asynchronous computing model, embedding the logic for triggering the next operation directly into a data packet's metadata. Once a packet finishes processing, it can independently decide what happens next. Being asynchronous generally gives you an efficiency edge over synchronous designs, since you're not forced to wait for every unit to stay in lockstep — GPUs, by contrast, are fundamentally synchronous. So Cerebras's claim of extracting extra efficiency through this technique is reasonable, though it's not a purely systolic dataflow design.

SambaNova, on the other hand, is a much more complete systolic dataflow design. The tradeoff: once model parameters are frozen directly into that dataflow graph — with no other feed path — the user's live attention data or KV cache has to flow through the entire engine end-to-end, and that KV cache size depends entirely on the user's context length. Since context length is inherently a user-defined variable, you can't have a truly fixed dataflow graph serve every possible context length efficiently — you end up needing multiple pre-configured graphs (say, one for under 100 tokens, a different one reconfigured for longer contexts), because repeatedly reloading model data defeats the entire purpose of keeping everything resident on-chip in the first place. Cerebras faces a related constraint since they only program one layer at a time — when the next layer loads, it has to be reprogrammed, and if a layer is too large, they likely have to split it across batches. Groq follows a broadly similar approach in building its own fixed dataflow graph.

Precision tradeoffs: fixed formats vs. GPU flexibility

Raminda: A lot of these architectures also have to commit to certain fixed, domain-specific math functions to save area — SambaNova, as far as I know, is fixed to FP16 for storage with FP32 for computation; that's a hardware-level decision, predefined and inflexible. If the user decides 8-bit precision would be good enough, you're stuck with whatever precision the hardware actually offers — whereas a GPU, because it exposes that flexibility in software, lets you do things like split a multiplier's precision, at some efficiency cost (multiplier cost scales roughly with the square of bit-width, while a simple precision split is closer to linear) — you lose some hardware efficiency, but you gain real flexibility.

I've honestly never been a big fan of simply "throwing more silicon" at a problem. But given how large today's models are, and how many chips are required regardless of architecture, the real question becomes how you efficiently integrate a large number of chips into a working solution. I've spent a fair amount of time modeling GPU architectures this way, and one metric I like using is tokens per second per square millimeter — a genuinely useful measure of silicon efficiency, since otherwise you could claim you'll process a billion tokens simply by pointing at an enormous physical footprint. The complementary metric is tokens per watt. At the end of the day, you want the best combination of both.

Wrap-up

Sunil: I think we're out of time — Sam, anything else you want to add?

Sam: Just that this was a genuinely illuminating conversation — covering Cerebras, Groq's LPUs, and SambaNova's reconfigurable dataflow units. Really appreciate you joining us, Raminda.

Raminda: My pleasure — thank you for having me, and hopefully we'll get to do this again.

Sunil: We'll have to catch up again soon and keep this conversation going. Thanks, everyone.

Sajjad: Thank you, everybody.