Episode 4 — Open Source Catches Up: Why Open-Weight Models Are No Longer a Compromise

Setting up today's topic

Sunil Baliga: Token Drop, week four. I've been thinking about two topics we could cover: open source and open-weight models, and world models. Both are big enough to deserve their own episode, so let's tackle open source and open-weight models this week, and save world models for next week.

On the open-source side — I was talking to a friend at a large company recently about their team potentially adopting open-source models, which I found genuinely interesting. I get a daily email digest from VentureBeat, and I recently saw a couple of notable items: someone ran the informal "pelican riding a bicycle" SVG-generation benchmark against Claude Opus 4.7 and Qwen 3.6, an open-source model — and Qwen actually beat Opus, running locally on a MacBook Pro. I also saw a piece from Databricks with the headline "Architecture Beats Model Quality."

Between that and the conversation with my friend, I'm increasingly convinced the market is going to mature into distinct segments. Some applications genuinely need the best, most capable LLMs available. But others — like our own experience running the FACTS benchmark, where an older model actually outperformed the newest generation — are perfectly fine running on non-leading-edge models. There's a growing field of open-source and open-weight options now: GPT-OSS (which I believe is open-weight rather than fully open-source), DeepSeek, Kimi K2, Qwen, and others. So I wanted to talk today about whether there's a real role for enterprises to use these open-source and open-weight models for certain classes of applications — not everything is a nail just because you happen to be holding a hammer, and I don't think that's true in AI anymore. What do you two think?

A brief history: from Llama to true open source

Sajjad Khazipura: For a long time, Llama was really the open-weights model to know — Meta released it as an open-weights model you could download and run yourself.

Sunil: We looked at it back then — it was a genuinely good model at the time.

Sajjad: Right, Llama was the leading open-weights model for a while. Then the Allen Institute for AI in Seattle — founded by Paul Allen — released OLMo, and specifically called it an open-source model, meaning something more expansive than open weights: you got the training datasets, the training scripts, and all the source code needed to rebuild the model yourself if you wanted to. That was their definition of true open source.

Since then, a lot more models have followed — most notably DeepSeek, which released DeepSeek-V3 back in January and genuinely set the world abuzz with its ability to build near-frontier-class models at a significantly reduced cost. They innovated across multiple areas to build meaningfully lower-cost models. They were open-weight, and also promoted as open-source — I haven't personally verified whether the full source code is available, but you can certainly download and use the open weights however you like.

DeepSeek was followed by Qwen, and — just this morning, in fact — another release: DeepSeek's newest model, which is near-frontier-class, built on a mixture-of-experts architecture with roughly a trillion total parameters. With mixture-of-experts, only a subset of parameters — around 50 billion — is actively used for any given inference, meaning you could realistically run it on a single large GPU, or maybe two, not much more. These large models have essentially become commoditized to the point where you can run them yourself, fine-tune them to fit your specific application, and retain full control — which matters a lot if you're running a commercial service. This is a genuinely interesting trend, and it's clearly challenging the bigger frontier labs like OpenAI and Anthropic.

I've mostly been reviewing benchmark results rather than running these models hands-on myself so far, but I'd like to actually deploy some of them in our own environment soon and see how they really perform. That's my overall read — and it's setting a broader industry trend: even Microsoft and Google are now releasing open-weight models, like Google's Gemma (Gemma 4 is supposed to be quite capable) and Microsoft's Phi model family.

An echo of the early Linux era

Sajjad: This whole dynamic reminds me a lot of the early days of Unix — proprietary Unix operating systems, and then a wave of open-source tooling around GNU.

Sunil: Right — and then suddenly there was the kernel.

Sajjad: I remember people used to ask "which version of Linux are you running, whose Linux is it?" The original Linux kernel release was a huge community effort, even though it was honestly barely good enough at first — it wasn't the best Unix-like operating system out there. But because it was freely available, adoption picked up quickly. Companies like IBM got involved early, contributing some of their own proprietary Unix capabilities into the open-source ecosystem and helping harden Linux for enterprise use. That was a genuinely pivotal moment — and today, the vast majority of the public internet runs on Linux, far more than on any proprietary alternative.

Sunil: I think this is a good development for the industry and for customers generally, because now there's real choice — it's not one-size-fits-all. There might be an open-source model that's much cheaper and perfectly suited to a given application; or maybe the application genuinely needs the most advanced model available. There's a wide enough range of applications now that not everyone needs the best, most expensive model all the time — even within a single application, some parts might reasonably run on an older or smaller model.

Sajjad: Right — and almost all of these are now being marketed as reasoning models. I haven't had a chance to really experiment with the open-source options yet — we're still mostly on commercial technology — but we're looking forward to testing open-weight and open-source models against our specific use cases.

The benchmarks: open models are genuinely competitive now

Sunil: Sam, you tinker with GPUs quite a bit — run models locally and so on. What's your take?

Sam Pooni: Twelve months ago, choosing an open-weight LLM generally meant accepting something like a 15–20% capability gap compared to the proprietary frontier models. People accepted that gap for specific reasons: data sovereignty concerns, cost control, and deployment flexibility.

Sunil: What do you mean by data sovereignty, exactly?

Sam: Essentially, where your data physically resides — think GDPR-style concerns. Companies were willing to accept a capability tradeoff specifically to retain full control over their data and infrastructure, exactly what Sajjad was describing with open-weight and open-source models.

Here's what I've been observing, though: just in the month of April alone, four major open models were released. GLM-5.1 from Z.ai came out April 7th, scoring 58.4% on SWE-bench Pro — edging out both GPT-5.4 (57.7%) and Claude Opus 4.6 (57.3%). Then Kimi K2 from Moonshot AI, released April 20th, scored around 80.2% on SWE-bench Verified, sitting only about 0.6 points behind Opus 4.6 (I'm aware 4.7 is out now and I've been using it, but I'm comparing against 4.6 here for consistency). Then DeepSeek's newest release, out today, April 24th, scored 80.6% on SWE-bench Verified — at roughly one-seventh the token cost of Claude Opus. That's a 1.6-trillion total-parameter model, with about 49 billion active parameters, released under an open license. And finally, Qwen3, a roughly 27-billion-parameter dense model from Alibaba, released April 22nd, which outperformed the previous Qwen 3.5 generation specifically on coding benchmarks.

What I'm seeing is that this isn't really a binary choice between "open-source" and "frontier" anymore — these models are genuinely catching up, and in some specific benchmarks, actually surpassing the models they're being compared against. It really comes down to your specific use case now.

Sunil: Or market segment, the way I look at it.

Sam: Exactly. Open-source and open-weight models are no longer a compromise option by default. They're performing at frontier-model levels in certain cases — sometimes slightly behind, but we genuinely can't discount them anymore, because at least four of these recent releases are very strong, and some are beating frontier models outright in specific benchmarks. I think this is only going to accelerate, which in turn forces the frontier labs to keep pushing harder — it's a real race, and everyone's trying to keep up.

Sunil: That's exactly what my friend was describing — his company is now looking at open-source models for some use cases, not all of them. Just a few months ago, they weren't even on the table.

Four underlying trends: MoE, long context, reasoning, and native multimodality

Sam: Right, and building on what Sajjad said earlier — mixture-of-experts is basically the default architecture now, that's a given. But what's happening right now, even in open-source models, is million-token context windows. Sajjad, can you speak to that?

Sajjad: Sure — DeepSeek's newest release is reportedly discussing something close to a near-infinite context window. What that practically means is the model can work across a very large corpus without degrading in performance.

Sunil: Wait, I thought a bigger context window generally meant more risk of hallucination?

Sajjad: That's generally true, and it comes down to how the attention heads are architected and how information is represented internally. What they're doing now is converting information into latent-space vectors for the attention heads' KV cache — so rather than storing raw tokens, the KV cache holds compressed latent-space vectors instead, at significantly lower dimensionality. That allows far more efficient memory use — they're claiming something like 5–13% of the memory footprint it would otherwise require. On top of that compression, innovative attention-head architectures are reducing computational overhead further — techniques like tiling in matrix multiplication to reduce cache thrashing, processing these vectors within a GPU's actual cache lines or cache windows rather than constantly evicting and reloading. Essentially, a lot of classic computer-architecture and systems-engineering techniques are now being applied directly to building these large models at meaningfully lower cost.

Why scarcity drove Chinese labs to innovate

Sajjad: As soon as DeepSeek came out, I remember hearing that Chamath Palihapitiya asked his entire team to spend a full day studying DeepSeek's optimization techniques and understanding why similar approaches hadn't already been applied internally. What's interesting is that Chinese research labs, operating under GPU export restrictions and unable to access the highest-end chips, had to make do with lower-end GPUs — which is exactly what forced them to develop these innovative optimization techniques in the first place. What I'm also learning now is that some of these labs aren't just working around lower-end GPUs — they're beginning to build their own custom silicon specifically for training these models. That's a genuinely interesting space to watch.

Native multimodality and new attention architectures

Sam: Building on that — you raised a great point about these models increasingly becoming reasoning models by default. We're also seeing multimodality shift from something "bolted on" after the fact to something natively built in from the start.

Sunil: What do you mean by "bolted on"?

Sam: Historically, multimodal capability was fused in fairly late in the training pipeline — a separately trained vision component fused in near the end, closer to inference time. Now there's "early fusion," where multimodal training happens much earlier in the overall training cycle itself, rather than stitched on afterward. That's becoming standard. So the four major trends I'm tracking in open-source models right now are: mixture-of-experts architectures, million-token context windows, configurable reasoning, and native multimodality.

Another interesting detail: GLM-5.1, the model I mentioned earlier, was actually trained on non-NVIDIA hardware — which has real implications, since it signals genuine diversification in the underlying hardware architecture landscape, echoing what Sajjad mentioned earlier. There's also a wave of new algorithmic innovation around long context specifically — sparse attention techniques, including some heavily compressed attention approaches recently introduced in newer releases, delivering something like a 27% gain in FLOPs efficiency and roughly a 10% reduction in cache memory usage. Even incremental-sounding improvements like that add up to something significant at scale. Open-source models catching up to, and in places matching, primetime frontier models is a genuinely big moment.

Sunil: And even where they haven't fully caught up, that's fine for plenty of applications — it's perfectly acceptable in a lot of cases.

A preview of next week: world models

Sajjad: You mentioned world models at the start of the call — a lot of these same models are now increasingly being presented as world models too, including Gemini 3.5 and Opus 4.7, largely because of their multimodal training. There are also more specialized efforts — World Labs, founded by Fei-Fei Li, the Stanford professor whose team built ImageNet and did a lot of foundational image-processing work years ago. Her lab is now focused on training models on video content specifically, in pursuit of genuine understanding of how the physical world operates — what happens when you drop a ball, for instance — with the hope that these models implicitly learn the laws of nature, like gravity, simply by observing millions of hours of video.

The real challenge: choosing among so many models

Sunil: With this many choices available now and more coming, it seems increasingly important to have solid benchmarks so customers can actually evaluate their options. We've mentioned a lot of models today — Kimi K2, Qwen —

Sam: It's not just about picking a model, Sunil. Take a trillion-parameter model as an example — it simply can't run on a single GPU, since even a high-end GPU tops out around 80GB of HBM capacity. So you have to shard the model across multiple GPUs, which brings tensor parallelism into play — and that means real communication overhead between GPUs. It's a tradeoff at every level: do you hold the full model, or partial shards, and how do those shards communicate and aggregate their output? That's always been a hard problem, and the algorithms handling it keep getting more sophisticated.

Sunil: Putting myself in a customer's shoes — we've named a lot of models today. If I actually had to choose one, how would I know which to pick? There needs to be some real guidance, because nobody has time to individually evaluate every model out there.

Sajjad: The capability surface of these systems is so vast that it's genuinely hard to evaluate competence across every dimension. There are literally hundreds of benchmarks out there — Humanity's Last Exam, GPQA, MMLU, and plenty of domain-specific ones — and these models seem to ace most of them. I remember Stanford aggregated a large number of these benchmarks under something called HELM, run by Percy Liang, a Stanford professor I know — they were evaluating leading models against this suite on a weekly basis. But it eventually became prohibitively expensive to run every benchmark against every leading model, simply because both the number of benchmarks and the number of models kept growing.

That said, benchmarks still have real value — if you're focused on a specific class of application, or a sufficiently narrow domain, there's a good chance a relevant benchmark already exists for that domain. You can either run it yourself against candidate models, or find that someone's already published results, which becomes a genuinely useful guide for selecting the right model for your application.

From "which model" to a routing problem

Sam: That's a great point, Sunil — this has really stopped being a "which model" question. It's becoming a routing and procurement problem, not fundamentally a capability problem. You have many different use cases, each suited to a different model, so the real challenge becomes matching the right model to the right workload.

Sunil: Right — and even within a single application, you might reasonably use several different models depending on the specific requirement.

Wrap-up

Sunil: I think we've hit our limit for today. Anything else you two want to add?

Sam: Really interesting discussion.

Sunil: Absolutely — that's what I love about this industry, there's always something new to dig into.

Sam: Every single day, something new.

Sunil: Exactly. Next week we'll get into world models, which should be just as interesting. Alright, talk to you both soon.

Sam: Thanks, guys — bye.

Sajjad: Thank you.