The AI scaffolding layer is collapsing. LlamaIndex's CEO explains what survives.

The scaffolding layer that developers once needed to ship large language model (LLM) applications is crumbling. Indexing layers, query engines, retrieval pipelines, and carefully orchestrated agent loops are no longer necessary. According to Jerry Liu, co-founder and CEO of LlamaIndex, this collapse is not a problem, but rather the point.

'As a result, there's less of a need for frameworks to actually help users compose these deterministic workflows in a light and shallow manner,' Liu explains in a new VentureBeat Beyond the Pilot podcast. Liu's LlamaIndex is one of the foremost retrieval-augmented generation (RAG) frameworks connecting private, custom, and domain-specific data to LLMs. However, he acknowledges that these types of frameworks are becoming less relevant.

With every new release, models demonstrate incremental capabilities to reason over massive amounts of unstructured data, and they're getting better at it than humans. They can be trusted to reason extensively, self-correct, and perform multi-step planning. Model Context Protocol (MCP) and Claude Agent Skills plug-ins allow models to discover and use tools without requiring integrations for every one independently.

The agent patterns have consolidated toward what Liu calls a 'managed agent diagram' — a harness layer combined with tools, MCP connectors, and skills plug-ins, rather than custom-built orchestration for every workflow. Furthermore, coding agents excel at writing code, meaning developers don't need to rely on extensive libraries. In fact, about 95% of LlamaIndex code is generated by AI.

'Engineers are not actually writing real code,' Liu said. 'They're all typing in natural language.' This means the layers between programmers and non-programmers are collapsing, because 'the new programming language is essentially English.' Instead of manual coding or struggling to understand API and document integration, developers can just point Claude Code at it. 'This type of stuff was either extremely inefficient or just would break the agent three years ago,' said Liu.

'It's just way easier for people to build even relatively advanced retrieval with extremely simple primitives.' So, what's the core differentiator when the stack collapses? Context, Liu says. Agents need to be able to decipher file formats to extract the right information.

Providing higher accuracy and cheaper parsing becomes key, and LlamaIndex is well-positioned here, he contends, because of its developments with agentic document processing via optical character recognition (OCR). Ultimately, Liu emphasizes the importance of modularity and agnosticism. Builders shouldn't bet on any one frontier model or overbuild in a way that overcomplicates components of the stack.

Retrieval has evolved into 'agent-plus-sandbox,' as he describes it, and enterprises must ensure that their code bases are tech debt-free and adaptable to changing patterns. They also have to acknowledge that some parts of the stack will eventually need to be thrown away as a matter of course. 'Because with every new model release, there's always a different model that is kind of the winner,' Liu said.