The missing step between hype and profit

In February, I stumbled upon a flyer at an anti-AI march in London that cleverly riffed on South Park's underpants gnomes. The flyer, produced by Pause AI, an international activist group, read: "Step 1: Grow a digital super mind. Step 2: ?

Step 3: ?" It ended with a plea to pause AI development until we understand what Step 2 entails. The reference to underpants gnomes originates from a 1998 South Park episode, where a group of gnomes steal underpants and present a pitch deck with a three-phase plan: "Phase 1: Collect underpants. Phase 2: ?

Phase 3: Profit." This humorous example has become a popular meme, used to satirize startup strategies and policy proposals. Elon Musk once invoked it when discussing his plans to fund a mission to Mars. Currently, it aptly describes the state of AI development, where companies have built the technology (Step 1) and promised transformation (Step 3), but the path to get there remains unclear.

Pause AI advocates for regulation as a crucial part of Step 2. However, the specifics of what this regulation should entail and who will enforce it are still debated. On the other hand, AI proponents are convinced that Step 3 will bring salvation and tend to overlook the middle step.

They envision a future with an "economically transformative technology," as OpenAI's chief scientist, Jakub Pachocki, described it. While they have a general idea of where they want to go, the route to get there is uncertain. Recent studies highlight the uncertainty surrounding AI's impact.

A study by Anthropic predicted which jobs will be most affected by large language models (LLMs), suggesting that managers, architects, and media professionals should prepare for change. However, these predictions are based on assumptions about LLMs' capabilities rather than real-world performance. Another study by Mercor tested AI agents powered by top-tier models on workplace tasks and found that they failed to complete most duties.

The wide disagreement on AI's future stems from various factors, including the motivations of those making claims and the limitations of current studies. Anthropic has a vested interest in promoting AI, and many predictions are based on the rapid progress of AI coding tools. However, not all tasks can be solved with coding, and LLMs have limitations, such as making strategic judgment calls.

When deployed, AI tools must work within existing workflows, which can be complex and contaminated with human factors. The lack of agreement on what is about to happen and how creates an information vacuum filled with wild claims and speculation. To bridge this gap, we need more evidence and transparency from model makers, coordination between researchers and businesses, and new ways to evaluate AI technology in real-world settings.