These startups are chasing the next big thing in LLMs
AI News Desk
·
MIT Technology Review
··
11 min read
MIT Technology Review ’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future.
MIT Technology Review ’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future. You can read the rest of them here .
Way back in the summer of 2017, AI researchers at Google put out a paper called “Attention Is All You Need,” in which they described a new type of neural network called a transformer. It proved to be very good at processing long sequences of data, especially text.
Nine years on, transformers are the engines inside every major large language model on the market. “The entire AI industry is built on transformers,” says Justin Dangel, cofounder and CEO of the AI startup Subquadratic. “They are one of the most important innovations in the history of computer science, and they’ve changed the world.”
But transformers are starting to show their age. Many of the recent advances in LLMs, such as the development of so-called reasoning models and their ability to handle large amounts of input at once, are not neat extensions of that core technology but workarounds that patch over some of its fundamental flaws.
A growing number of scientists and engineers are now asking what’s coming next. LLMs are not going anywhere, but the way they get built is up for grabs. ( MIT Technology Review dubbed this future generation of models LLMs+ in this year’s list of the 10 things that matter in AI .)
Enter a wave of startups hoping to push the boundaries of this boomtown technology. Some will no doubt fail—but they have everything to play for and far less to lose than the companies at the front of the pack today.
But first, the problem. The key strength of transformers lies in a mechanism called dense attention, which encodes the meaning of a block of text in a series of numbers . The process involves comparing every word (or part of a word, known as a token) in that text with every other word via a form of multiplication.
Dense attention can capture the meaning of text with remarkable accuracy. But as the length of that text grows, the number of computations needed to process it adds up fast. A document 10,000 words long might require a transformer to perform 50 million multiplications. That’s the main reason LLMs suck up so much power .
The costs are huge. OpenAI is set to spend $50 billion on computing this year , according to the company’s president, Greg Brockman. And the International Energy Agency predicts that the total amount of electricity consumed by data centers will double by 2030 .
What’s more, transformers struggle with what many of the latest models are designed to do. Because of the way they process text word by word, transformers are not great at keeping track of a lot of information at once (in other words, what’s known as their context window cannot get too large). And yet if LLMs are to carry out harder tasks, they will need to take in larger amounts of data: a whole library of documents, an entire code base, or in the case of agents, output from other LLMs.
As for reasoning models, they work by writing notes to themselves (in a kind of scratch pad known as a chain of thought ) and then reading them back, which again adds to the amount of data to stay on top of.
As LLMs get bigger and better, transformers have become a bottleneck. The technology’s key strength is now a limitation.
Here are four new ideas for how to solve the transformer problem—innovations that could change LLMs for good, making them faster, far more efficient, and (maybe) even smarter.
An obvious way to make LLMs faster and cheaper is to tackle the problem head on and change the way attention works. Swapping out dense attention for a mechanism called sparse attention, which runs calculations on only some pairings of words in a block of text instead of all of them, can radically reduce the amount of computation LLMs need to do.
Researchers have come up with plenty of sparse attention mechanisms over the years. The problem is that none of them were as good as dense attention at capturing meaning.
That might have changed. Subquadratic, a startup based in Miami, claims it has invented the first sparse attention mechanism that rivals top mainstream LLMs on a handful of tasks, including search and coding. It’s a huge claim (and some people in the industry remain skeptical ).
Subquadratic says its model, SubQ, works by figuring out on the fly—for each piece of text it is given—which words matter and which don’t. The company also claims that thousands have signed up to its waitlist and plans to make the model widely available soon.
Meanwhile, Manifest AI, a startup based in San Francisco, is coming at the problem from a different angle. Instead of changing how attention works, it is replacing it with something else.
It has developed a mechanism it calls power retention, which stores only the most relevant information for a given task and ensures that the amount of data an LLM has to keep track of doesn’t blow up.
Attention mechanisms force LLMs to keep track of everything in their context window. A sparse attention model (such as SubQ) throws out a lot of the individual words, but it still retains a rough picture of everything it has seen. In contrast, power retention works by providing the model with a rolling summary of its context window. As new information is added, less relevant information is dropped.
The basic principle of retention has been around for a decade. Manifest AI claims it has updated those techniques to build models that can stand up to transformer-based LLMs for the first time.
The company says it is possible to adapt a transformer model into a power retention model with minimal retraining. To demonstrate this, it has turned an existing open-source coding LLM called StarCoder into a version that uses power retention, called PowerCoder. It has also released a model called Brumby, which it claims rivals some versions of Alibaba’s popular open-source model Qwen.
Manifest AI wants its power retention tech to become the go-to solution when LLMs need to carry out tasks that involve processing huge amounts of data. There are many useful applications, Manifest AI’s cofounder and CTO, Carles Gelada, claimed in a video announcing his company’s technology last year—from analyzing videos that are hours long to building agents that can stay on task for weeks at a time.
Liquid AI, an MIT spinout based in Cambridge, Massachusetts, hasn’t changed or ditched transformers fully but pairs them with its own tech, liquid neural networks, to build what cofounder and CEO Ramin Hasani calls LFMs (liquid foundation models).
Liquid AI’s models are far smaller and use less energy than most LLMs. The firm builds models for car makers, including Mercedes, which run on the small chips inside vehicles. Its latest models can run on a Raspberry Pi, a low-powered hobbyist computer that costs $50.
Its models are available for free to any organization with an annual revenue less than $10 million. And they have proved popular: The company has racked up almost 34 million downloads, says Hasani.
Liquid neural networks were inspired by worm brains. They are an extension of another type of neural network that predates transformers, called convolutional networks. The key innovation is a mechanism that lets a model adapt its behavior to new information, so it can learn as it goes. That’s not possible with transformers: Once a model is trained, its behavior is fixed.
Liquid AI’s first models were pretty basic but could fly drones or drive vehicles. With LFMs, the company is trying to scale up its technology to compete with mainstream LLMs. Its new models match the performance of rivals four times bigger, including versions of Alibaba’s Qwen and Google’s open-source LLM Gemma.
A typical LLM is built from a stack of transformers wired together. Liquid AI’s recent LFMs are hybrid models made up of 20% transformers and 80% liquid neural networks.
That ratio was hit upon by another AI system that Liquid AI has built, which it uses to help design all its models. “It’s the core technology of our company right now,” says Hasani. This designer AI sifts through many different combinations of neural networks—liquid, convolutional, and more, as well as transformers—and comes up with designs that bolt different ones together to hit a sweet spot of performance and efficiency.
Hasani thinks transformers were just the beginning: “Your brain is an AGI system, you know, and it operates with 20 watts of power. How is it possible? We can get a lot more innovative.”
Almost all LLMs produce their output one word at a time. It makes sense, because that is how people speak and write. But for computers, it’s very inefficient.
It is faster and cheaper for LLMs to generate text all at once—spitting out whole sentences or paragraphs in one shot. That’s the approach taken by Inception, a startup based in Palo Alto, California, which is building LLMs using a technique called diffusion.
Diffusion is better known as the technology that drives most image and video generation models . Diffusion models are trained to take a random grid of pixels—like the static on an old TV set—and turn it into an image . They do this by working on all the pixels at the same time, figuring out which need changing to make the static look more like a high-definition photo.
It turns out this process works on text too. Inception has trained its LLMs to take a random string of words and turn it into sentences that make sense. Diffusion LLMs still use transformers to encode meaning, but by producing whole blocks of text at once, they make transformers do more for less. “You’re still using a big transformer model, but you can predict many tokens at the same time,” says Inception’s cofounder and CEO, Stefano Ermon. “That’s why these models are so much faster and cost-efficient compared to what most other people are building today.”
The challenge was to take a technology designed for image generation and apply it to text. With images, if you need to change a blue pixel to a red one you can step through intermediate colors, says Ermon. That doesn’t work with text: “When you have ‘cat’ and ‘dog,’ there is not really something in between.”
Ermon is also a researcher at Stanford University. In 2024, he and a pair of his Stanford colleagues figured out the math to make diffusion models work with text. They trained a diffusion model that matched the performance of GPT-2—an LLM that OpenAI built in 2019—but was 10 times faster. It was enough for Ermon to spin out a company.
Today he has his sights on the big league. Inception claims its latest model, Mercury 2, performs as well as some of OpenAI’s GPT-4 models, released in 2023, but again 10 times faster. “We’re bullish about this approach because it’s the one that is going to scale up,” says Ermon.
The only things that matter are speed and cost, he adds: “Ultimately, the currency is going to be intelligence per dollar.”
Inception is not the only company betting on diffusion. Google is also experimenting with this approach and has built a prototype LLM called Diffusion Gemma . But Ermon is not worried about the competition. “I think it’s validating,” he says. “This is the future.”
Pathway, another startup based in Palo Alto, is perhaps the most extreme of this new bunch. It wants to free LLMs from the constraints of language.
The firm has built a type of LLM called Dragon Hatchling (named after the dragons in Terry Pratchett’s novel Color of Magic , which materialize if you think about them hard enough). Its standout result so far is a high score on a benchmark that pits LLMs against more than 250,000 very hard sudoku puzzles. Dragon Hatchling beat more than 97% of the puzzles; several leading LLMs from the top labs failed to solve any.
The point Pathway wants to make is that despite their remarkable success at many different tasks, there are still crucial classes of problems where LLMs fail. Sudoku is just one example. If we want LLMs to come up with genuine, novel solutions to real problems, we need to move beyond transformers, says Pathway’s cofounder and CEO, Zuzanna Stamirowska.
That’s because transformers force LLMs to do everything with text. But language is not the best tool for certain kinds of reasoning. “It’s very difficult to represent a sudoku board word by word,” says Stamirowska.
Pathway’s solution is to change the math behind the transformer, replacing the attention mechanism with a mathematical structure called a state space. Instead of encoding information word by word, state spaces compress it into a more abstract representation. Using this technique, Dragon Hatchling can still process and produce text, but it can also mimic forms of reasoning that do not involve sequences of words. This not only makes Pathway’s model more efficient, but (in theory) it lets it take on tasks that other LLMs cannot do.
Think of chess or mathematics—those kinds of puzzles are not held in your head as a long sentence, says Stamirowska: “The eureka moment that pops up in your brain isn’t necessarily in language. We would argue that if you have to reason in language, you’re somehow constrained.”
Stamirowska admits that a mainstream LLM could read a book about how to solve sudoku and then write code to do it. But we want to build models with more than book smarts, she says: “The hope for AI is not to solve sudoku; it’s to cure cancer. There’s not a book for that.”
“Transformers are an engineering convenience that we fell on,” she adds. “It started a religion, but it’s silly to think that a breakthrough won’t happen again.”
