Guide to Loop Engineering: How ‘autoresearch’ and ‘Bilevel Autoresearch’ Turn AI Agents Into Autonomous Machine Learning ML Research Loops

Most people still use AI like a 2015 search box. You type, you read, you type again. A newer pattern replaces that manual back-and-forth with a loop. This guide explains loop engineering using two verified artifacts. The sources are Andrej Karpathy’s autoresearch repository and the Bilevel Autoresearch paper . The framing follows a write-up by @0xCodila .

To start, compare two modes. A prompt is one instruction, after which you decide the next step. A loop, by contrast, is a goal the model pursues until it arrives. The model plans, acts, checks its own result, then repeats. You define the objective once, and the loop handles iteration. Crucially, a loop only earns its cost when the work is measurable.

So what separates a real loop from a chatbot on repeat? Every reliable loop has three components.

These three parts are not theoretical. On March 7, 2026, Karpathy released autoresearch , an open-source repository under the MIT license. It ships three core files and about 630 lines of code. The project went viral within days and now sits near 90,000 GitHub stars. It was latter presented as the pattern “the Karpathy Loop.”

The design is deliberately small, yet strict. The agent edits only train.py , which holds the GPT model, optimizer (Muon and AdamW), and training loop. It cannot touch the evaluation utilities in prepare.py . That separation stops the agent from making the test easier instead of the model better. Meanwhile, a human writes program.md , the instructions the agent must respect.

Each cycle runs one experiment. The agent reads the code, proposes a change, trains five minutes, then keeps or rolls back. The scoring metric is val_bpb , validation bits per byte, where lower is better. That budget yields roughly 12 experiments per hour, so about 100 run overnight.

The reported outcomes are concrete. Karpathy pointed it at his already-optimized nanochat GPT-2 training code. It ran for two days and completed about 700 experiments, keeping 20 genuine improvements. Stacked together, those fixes cut GPT-2-quality training time by 11%, from 2.02 to 1.80 hours. One finding was a QK-Norm implementation missing a scalar multiplier, which had left attention too diffuse across heads.

Notably, humans tire after roughly a dozen experiments, whereas the loop does not. Separately, Shopify CEO Tobi Lütke ran autoresearch overnight on an internal model. He reported a 19% improvement after 37 experiments. Karpathy’s takeaway: if you have an objective metric, you are the bottleneck.

The differences become clearer side by side.

Consequently, AI engineering teams now assemble working loops from five reusable pieces :