IBM Built the Cold War’s Most Powerful Code Breaker for the NSA

Most Powerful Code Breaker for the NSA">

At the height of the Cold War, one very specialized computer was so secret that the world didn’t know it existed. It ran its jobs up to 200 times as fast as any other computer of its time. It was the U.S. National Security Agency’s main cryptographic processor in operation from the time of the Cuban Missile Crisis in 1962 through the Vietnam War and on past the 1975 Helsinki Accords . The machine stopped running only when its moving parts finally gave out. The Harvest computer mattered because of what it was as well as when it ran. For 14 years, it was the engine processing the NSA’s most sensitive intercepts at a time when signals intelligence was as close to a strategic weapon as anything short of a warhead. Designed and built by IBM for the NSA, Harvest was one of the first machines designed to apply operations to enormous datasets rushing past, a precursor to the computers today that manage continuous video streams and security systems in real time. It was also one of the first machines built as an add-on —a specialized helper intended to do one job exceptionally well, bolted onto a general computer. Harvest’s modular design is like a 1960s version of today’s graphics chips that CPUs use to run intensive video-game and AI processing loads. All that raw processing power meant that Harvest also needed nonstop rivers of data to run on. And that led to another pioneering achievement: the world’s first automated tape library that could robotically fetch any one of hundreds of large cassettes of magnetic tape from the machine’s racks. Given Harvest’s unprecedented processing and storage capacity, the machine’s designers naturally needed to rethink how their system handled information. So IBM wrote a customized programming language called Alpha to let code breakers rigorously describe cryptographic problems, just as scientists at the time were using the emerging language Fortran to describe equations and data-processing algorithms. In Fort Meade, Md., an NSA data center hosted one of the world’s fastest computers of its time—although not often discussed, because of its sensitive, high-security code breaking and cipher hunting work. National Cryptologic Museum The story of Harvest, pieced together from declassified documents and contemporary manuals and technical overviews , provides a new and unexpected vista on the history of computing. It also offers a case study in how national security needs, especially during the Cold War, pushed computer technology beyond the far reaches of what unclassified, civilian computing could achieve. Harvest’s distinctive history reveals a visionary algorithmic, coding, memory, and hardware architecture occasionally decades ahead of its time. But this machine was also built only once, for one singular purpose, and then ultimately quietly retired. The Heart of NSA’s Secret Machine IBM’s landmark 1960 transistorized mainframe, the IBM 7030 , better known as Stretch, provided the front end for Harvest (which was officially known as the IBM 7950). IBM delivered Stretch to eight or nine customers , mostly scientific research labs, from 1961 through ’63. Designed and prototyped throughout the second half of the 1950s , Stretch introduced the now standard notion of an 8-bit byte. For its first three years of operation , Stretch was the non-classified world’s fastest computer, although it failed to meet IBM’s aggressive goal of running 100 times as fast as Stretch’s predecessor, the IBM 704 . While IBM engineers in Poughkeepsie, N.Y., were designing and building Stretch, the company was also quietly discussing a new system that would be built for NSA. At the time, NSA’s existing cryptanalytic computers—large, batch-processing machines that required human operators to manually stage each tape run—were struggling to keep pace with the sheer volume of intercepted message traffic coming in from around the globe. What the agency needed was a machine that could process an unbroken river of incoming data, automatically, around the clock. That requirement alone profoundly shaped Harvest’s design. IBM’s Harvest system, custom-built for the NSA for code breaking, paired the IBM 7030 Stretch mainframe with a bespoke data-stream processor. Stretch handled ordinary computing and input/output, including the Tractor automated tape library. Both units shared two kinds of memory: a large main bank and a smaller, faster bank. When Stretch switched to streaming mode, Harvest drew two streams of data, P and Q, from memory, processed them in parallel, and returned the results as a third stream, called R. Chris Philpot After two failed proposals to NSA, in 1958 IBM finally landed the contract: a Stretch-based machine, augmented by a custom coprocessor, with a revolutionary tape-based storage system, called Tractor. Stretch’s forte was floating-point math for scientific computations. IBM had designed it primarily for labs working on frontier research like nuclear weapons design and weather prediction . By contrast, the custom coprocessor to be built atop Stretch would help NSA analysts sift through alphanumeric characters—that is, essentially integer data. Harvest’s coprocessor was the opposite of a general-purpose system. It was, rather, a streaming computer . Instead of executing long series of instructions, it followed one fixed sequence of steps and applied that same sequence to every pair of characters as they streamed past. Harvest shared memory with the main Stretch processor and ran in bursts. Either Stretch was operating, or else it suspended itself while Harvest’s coprocessor shot through data in memory at extreme speeds. Stretch and Harvest were among the first large computers built entirely from transistors packaged in circuit cards and housed in large, refrigerator-size frames. A 1962 technical manual about Stretch describes the machine’s CPU as divided into functional sections—the instruction unit, the look-ahead unit, the (parallel and serial) arithmetic unit, and the memory bus unit. Harvest inherited Stretch’s basic circuit design but then added something unconventional: Its streaming units processed data in overlapping stages called a pipeline. So while one pair of data bytes was being compared, the next pair was being fetched from memory. Harvest’s coprocessor operated by fetching two streams of data, called P and Q, from the system’s memory, performing operations on them, then writing the results to memory as a third stream, R. Each stream could be anywhere from 1 to 8 bits wide. Harvest’s memory was bit-addressable, meaning word boundaries could be ignored entirely. For instance, it could fetch just 5 bits rather than filling out a whole byte. Streams P, Q, and R included flexible provisions for looping and addressing data in complex patterns—allowing, for example, repeated fetching of short strings from memory. Data from P and Q fed into two functional units. The simpler was the logic unit, which performed basic, bitwise operations—the same operations any programmer would recognize today—and wrote its results back to memory. The more complex was a table-lookup unit. It combined incoming data from P and Q to form an address in memory, which could then be used to advance a counter by one, set a specific bit, or retrieve a stored value. The latter unit functioned, in effect, like the rotor wheel inside a cipher-encoding/decoding machine of the era, the kind that electronically substituted one value for another according to the cipher machine’s wiring. Harvest’s complexity baffled some at the NSA. During employee tours, according to James Bamford ’s 2001 NSA history, Body of Secrets (Doubleday), officials would point to the machine and scoff, “It’s beautiful, but it doesn’t work.” Not everyone at the agency was put off by the monumental device, however. One of the few documented examples of Harvest at work, recounted by Bamford, describes the machine searching 3.5 billion characters of text for any of 7,000 target terms, in just under 4 hours. In unclassified remarks from 1972, NSA analyst Robert Looney mentions one job Harvest had tackled—though he didn’t specify the end goal or the code-breaking effort behind it. Codenamed “Moretown,” the job involved sifting through 11 million messages spanning 16 years of intercepted traffic against a list of some 8,000 search terms—all in about ten hours. IBM’s Frances Allen helped design Alpha, Harvest’s custom-built programming language. IBM IBM’s James H. Pomerene was chief engineer of Harvest, supervising its custom-designed circuits that’d been optimized for algorithms used in many cryptographic jobs. IEEE IBM’s Fred Brooks Jr. was a key co-architect of Harvest’s hardware system. Computer History Museum As a unified system, Harvest—that is, Stretch plus IBM’s custom-built streaming processor add-on—streamed 1 byte every 0.3 microseconds, and it boasted about 800 kilobytes of addressable memory. “Here you see one bank pulled out of its oil bath,” Looney said in his 1972 remarks celebrating Harvest’s tenth anniversary of operations. He held up a photo of Harvest’s magnetic core memory banks—six of them, submerged in oil for cooling. Factor in the time demands of various data fetches from Tractor’s tape archives, and a single Harvest “instruction” sometimes carried on, without needing any human intervention, for hours. “It was quite an amazing computer,” recalled IBM Fellow Emerita Frances Allen in a 2001 oral history . “One instruction, for example, could do sorts, and do statistical analysis of the data that was streaming by it.… Everything we were doing at that time was on the cutting edge. There was no question about it.” Allen, who received the A.M. Turing Award in 2006 , was one of the developers who worked on both Stretch and Harvest. At the time she started working on Harvest, Allen noted, the Fort Meade, Md.–based NSA was largely unknown outside of classified intelligence circles. So she at first assumed she was working on an unspecified naval project. “We thought of ourselves as working for the Bureau of Ships, because that was the code name for NSA in the budget!” recalled Allen, who died in 2020. Other key Harvest designers and early developers wound up becoming influential figures over the course of computing history. Frederick Brooks Jr. , recipient of the 1999 Turing Award and a major contributor to the hardware and software for IBM’s System/360 , also helped develop Harvest. And James Pomerene , prior to his involvement with Harvest as its chief engineer, had previously helped build the pioneering IAS computer alongside John von Neumann. How Tractor Stored a World of Data IBM built the Tractor tape system ( IBM 7955 ) to attach to the same Stretch machine that hosted Harvest, because no existing data storage technologies could keep up with the computer’s staggering throughput. Stretch handled the business of staging tapes from the library to the drives—using Tractor’s automated cassette handler. Stretch also coordinated reading data in from Tractor and writing results back out from Harvest. Harvest, in turn, did all its actual computing on the system’s shared main memory. In the early 1960s, and even after Tractor and Harvest were installed, hard-drive data storage was in its infancy. For code-breaking jobs of the size Harvest was taking on, disk storage would have been impractical in terms of both cost and sheer floor space. So Tractor had to be based around tape storage. Each tape was sealed inside a case built like a boombox—twin encased reels under a window, carried by a handle—and, at 6 to 7 kilograms, about as heavy as a bowling ball. Think of a Tractor cassette as an outsize predecessor of the audiocassette, which would come along a decade later, and holding some 120 megabytes of data on a reel of tape 550 meters long. Each storage unit housed up to 160 of these cassettes. An IBM technician holds one of the data cassettes used with Harvest’s automated Tractor tape drives. IBM When Harvest launched in 1962, it had three automatic cartridge units, each serving two drives. So the available online storage across the three Tractor units totaled a stunning 44 gigabytes. That’s more than 190 times as much capacity as the IBM 2314 disk storage system, announced in 1965, which held 233 megabytes across its full complement of eight drives . Tractor had to run continuously, swapping cassettes in and out, 24 hours a day, seven days a week. The system’s tape-handling speed was tuned to keep pace with Harvest’s own appetite for data. The custom-built robotic mechanism for retrieving the cassettes was a servo-driven arm that traversed the system’s storage racks. It fetched a cassette from its slot and delivered it to a handler or received a cassette from one of the handlers and returned it to storage. Running at 6 meters per second, Tractor’s tapes zipped past the read/write heads faster than the eye could track. Software running on Stretch handled the cassette shuttling as well as reading and writing. For one of Tractor’s drives to move from the completion of processing one tape to reading the next took about 18 seconds, assuming it had already been fetched and was ready to mount. Robotically fetching a cassette from the storage unit and preparing it for reading required no human handling or input whatsoever. In addition to Tractor, the system had standard reel-to-reel tape drives attached to Stretch. Harvest’s technicians often used the conventional drives for importing and exporting data to and from other systems; there was no other practical way to get large datasets into or out of Harvest. Tractor could also store permanent files and retrieve them directly from its tape libraries when a job required them. In other words, Tractor’s substantial cassette libraries acted both as permanent data storage and as a place to hold transient data for processing by Harvest. No system in the commercial computing world of 1962 came close to Tractor’s gigabytes of simultaneously accessible data. At most computer centers at the time, “available” data meant physical racks of tape standing somewhere near its drives—accessible only as rapidly as an operator could manually pull a reel and thread it onto a machine, one at a time, over the course of a shift. Alpha Was Harvest’s Custom-Built Programming Language Created jointly by IBM and NSA, the Alpha language existed solely to program Harvest’s streaming dataflow engine for code-breaking work. According to a declassified Pentagon history of NSA computers , Alpha stood for Advanced Language for Programming Harvest. Alpha allowed the programmer to define the alphabet in which code-breaking data would be processed. The language also included two unusual characters with no equivalent in conventional computing until years later, when Multics and Unix introduced wildcard characters. A “scab” (which was represented on Harvest’s input keyboard, a repurposed early IBM Selectric typewriter, by a “?”) stood for a character that was real but unknown. And a “pad” (represented by a blank space) was a null or spacer. These characters provided flexibility of representation for code breaking jobs, in which unknown or uncertain characters were commonplace. A Harvest operator types on one of the main system consoles, a repurposed IBM Selectric typewriter. IBM The rules governing Alpha’s operations on strings anticipated other modern rubrics, like “ not a number ”—a designation describing an unknown value in a dataset that can propagate through calculations, rather than silently corrupting them. Strings in Alpha could also be aggregated into cords, and cords into ropes, giving cryptanalysts a hierarchical vocabulary for describing complex intercepts. Allen wrote a final technical report on her section of the Harvest software when her part of the project concluded—and just as promptly lost access to it. “I spent the good part of a summer on that,” she recalled in 2001. “And it just disappeared into Fort Meade somewhere.” Replacing an Irreplaceable Machine By 1971, according to NSA analyst Looney, the machine was running at its highest utilization ever—115 hours of production a week, or more than two-thirds of the time. Yet the number of jobs it processed had been dropping since 1967. Ordinary data-processing work, Looney noted, was by 1972 migrating to newer, general-purpose machines, leaving Harvest to concentrate on the very large, specialized jobs no other system could handle. At its tenth anniversary of operations, Looney concluded, Harvest was a machine “conceived in the fifties, born in the sixties, and irreplaceable in the seventies.” He got the last part wrong. On 27 February 1976, operators shut down Harvest for the last time. A custom mechanical component in the Tractor tape library had worn out, and the manufacturer of the part was no longer in business. By then Harvest had run continuously for nearly a decade and a half—through the roughest close call in the history of mutually assured destruction and into the age of détente—processing intercepts at a rate no civilian machine could touch. By the time it retired, Harvest had outlived several generations of commercial computing. A placard commemorates the 1976 decommissioning of IBM’s Harvest computer at the NSA’s headquarters in Fort Meade, Md. National Cryptologic Museum Somebody at the NSA decided to commemorate the machine with a mock telegram, written under Harvest’s name on the machine’s last day (and now preserved in the agency’s archives). “I first began operations at NSA. Although not widely known, I was probably the largest, fastest, and most technically advanced computer system in the world,” the telegram said. “And now, fourteen years later, the time to retire has come. The cost of my upkeep and operation has been overtaken by more modern equipments and the newer technologies.” The NSA ultimately replaced Harvest with the landmark Cray-1 supercomputer. The Cray-1 was built from faster, more tightly integrated circuits that could outperform Harvest’s aging transistors at nearly any task, including text processing. Although the Cray was designed primarily for numeric and scientific computing, it sold across many fields—which ultimately made the supercomputer win out once Harvest’s custom-built text-processing hardware was no longer worth the upkeep for just one customer. The secrecy that shrouded Harvest meant it could claim no lineage of immediate successors. But the ideas it pioneered didn’t disappear—they resurfaced, again and again, in the years that followed. Tractor’s automated tape library was the forerunner of the robotic storage silos that would become standard in enterprise data centers about 20 years later . Harvest’s pipeline architecture prefigured the dataflow computing movement of the 1980s . The continuous pattern-detecting logic of its match units finds direct echoes in modern hardware packet-inspection intrusion detectors and programmable network switches that today route traffic through the internet at wire speed. Harvest didn’t found a dynasty. But, in its time, it steadfastly pointed toward the future—in several directions at once.