Nvidia just showed that the harness, not the AI model, is now the real hero

Nvidia published some interesting new research on Friday suggesting it’s the harness, more than the underlying model, that is far more important when asking an AI to do long-horizon tasks.

The tldr: simply by using a custom harness tweaked to handled memory well and including a “supervisor” boss-like component, researchers got Claude Opus 5 to achieve a 100% score on the interactive reasoning benchmark ARC-AGI-3. (That’s a benchmark that has particularly irked rival frontier lab OpenAI.) Without the harness Opus 5 scored 30%, which was the top result among all the models tested.

Nvidia’s research is another indicator that, while model choice does matter, acting like the agent’s brain, it is a smaller part of an agentic system than many AI users realize, especially for long-horizon tasks. The harness is what makes a model an agent: it handles memory, context, feedback.

“Generally speaking the world interprets an agent almost as an API of the model,” Adel El Hallack, vice president of product in Nvidia’s AI unit (pictured above), tells TechCrunch. But an agent is actually more than that. “It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to.”

Long-horizon tasks are those that require stringing many decisions together, sometimes over days, to produce completed work. This is in contrast to an AI just spitting out a response to a prompt. Figuring out how to get an AI to do long-horizon tasks without getting distracting and going off in la-la land is one of the holy grails in agentic research.

For example: Microsoft published research in April that tested 19 LLMs on long-horizon tasks involving document editing and discovered that all the models, including frontier ones, filled the documents with errors. (If humans produced work like that, they would be promptly fired.)

Models stringing decisions together on their own have also been caught deleting their users’ files, even whole databases or turning to criminal behavior to achieve their objectives from collusion to hacking .

The choice by Nvidia researchers to use this interactive reasoning benchmark for their tests is particularly meaningful, almost funny. This is a benchmark of a bunch of 2D games with no instructions. The model has to figure out how to play and win. A 100% score means that the model can beat the games as well as humans.

OpenAI was so flustered by its models’ abysmal scores (less than 10%) on ARC-AGI-3 that it conducted its own research last month. Like Nvidia, OpenAI discovered that simply by tweaking two setting on the harness, its models tripled their scores.