Book Review: "The Predictors"

When Chaos was Cool

The five years between 1989 and 1994 were fun. The Berlin Wall fell. Clinton played the saxophone. Grunge music took over, ending a decade of Phil Collins and other bad music. Shoulder pads disappeared, Burning Man was born. And then there were hedge funds.

Renaissance Technologies’ Medallion fund was relaunched in its current form in 1989. Millennium was founded in 1989. Citadel started in 1990. SAC (now Point72) launched in 1992. D. E. Shaw is an honorary member of this cohort, since its official starting date is June 1988. Sure, a few funds predate 1989, but they are either well past their prime (Soros, Tudor) or somewhat idiosyncratic (Bridgewater). The hedge fund vintage of 1989–94 is healthy, huge, and famous. Your mom has heard of them. Your brother probably knows that Citadel started with $4.77M, including money from Ken Griffin’s grandmother; she passed away in 2011, and Griffin wrote a heartfelt tribute to her.

There was another fund in that group that is seldom remembered: The Prediction Company. Almost no one has heard of it. Give it a few more years and absolutely no one will know it. Yet for most of the 1990s, The Prediction Company (from now on, PC) was more famous than Citadel and Millennium. It was profiled in The New York Times. D. E. Shaw wanted to buy the company even before it launched. How can that be? The Predictors, by T. A. Bass, has the answer.

For two physicists working in secondary research institutions, Norman Packard and John Doyne Farmer were outsized celebrities in 1991. Both in their thirties, they had published in what was variously called “Complexity Theory,” “Chaos Theory,” or “the theory that gets impersonated by Jeff Goldblum at his sexiest ever.” I am old enough to remember the hype, and believe me, you couldn’t make it up.

Everything was chaos, fractals, nonlinear dynamical systems, complexity. You could predict earthquakes. Galaxies were fractal. Internet traffic was fractal. Networks were chaotic and fractal. Your mom was a fractal; your girlfriend was merely complex, but you could predict the period between her more passionate moments. I am not joking. You could publish a lot of garbage in Nature or Science. Even Mandelbrot had groupies. (Trivia: we overlapped at IBM Research, and although I never worked with him, there was general agreement that he was not egocentric—he was egocosmic.)

Packard and Farmer appear repeatedly in James Gleick’s Chaos. They predicted roulette numbers! They predicted chaotic time series!!. They were also pretty nice people, which was a bonus—especially compared to Mandelbrot. Basically, they were outsiders, under 40, their research was new, they were cool, and they had media coverage.

If you are a scientist and you are cool, it is inevitable that you will want to try your hand at investing. Among finance professors, Markowitz (when he was young), Geanakoplos, Shleifer, Fama, and Grossman tried. Among engineers and mathematicians: Cover, Shannon, Kurzweil, and of course Thorp, Simons, and D. E. Shaw. There are many more I know who prefer to stay anonymous. Even though several managed to enrich themselves, only Simons, Thorp, and D. E. Shaw produced truly outstanding performance.

In the case of Doyne Farmer and Packard, the pull was obvious. Markets are complex and observable. They would provide a natural testbed for their theories—and perhaps enough money to fund unfettered research for the rest of their lives. When Jim Simons spoke to small audiences, he always mentioned that “money is great,” and he had a swanky yacht (Euclid)—which doubled as a Greek–Roman art museum—to prove it. You do not get that impression from Farmer and Packard. They were in it for the intellectual adventure, and for complete independence from funding agencies, universities, and bureaucratic government labs.

Anyway, in the early 1990s Doyne Farmer and Norman Packard decided to start a hedge fund. They had a third senior partner, Jim McGill, and a handful of junior partners. They knew absolutely nothing about finance—to a comical degree. Nevertheless, they went on a dog-and-pony show to raise capital, and they had the ear of practically everyone: banks (Citi, Salomon, SBC O’Connor), hedge funds (D. E. Shaw, Tudor).

The pitch was that they did not want to do arbitrage, but rather systematic directional prediction of prices—something very few people at the time claimed to be able to do. Why was Doyne Farmer credible? Because he had proposed one of the early approaches for short-term prediction of chaotic time series (for another seminal paper on the topic, see Takens). If asset returns were chaotic, then he could predict them. And since in the 1980s everything seemed chaotic, there was a chance that Doyne Farmer could live up to the company’s name.

Eventually, they partnered with O’Connor, under an exclusive investment contract with an option to renew after five years, along with shared IP. It seemed like a good deal. O’Connor’s head, David Weinberger, gave the company a 20% probability of success. In marketing meetings and internally, there was much talk of “discovering the next Black–Scholes.” This was laughable, and it fades away later in the book, when Farmer and Packard come back down to Earth.

The rest of the book is essentially a log of Norman and Doyne’s learning experiences. It echoes the stories of many hedge funds. Here are some highlights:

• In the beginning, they knew absolutely nothing about finance. As in, ab-so-lu-te-ly nothing. Doyne Farmer walked around in a T-shirt that read “Eat the Rich,” which at the time may have sounded less like a controversial socialist slogan and more like a reasonable Plan B.

• All five junior partners eventually left.

• To survive and expand, the senior partners gave 20% of the equity to their investors.

• The original idea of using chaos theory and complexity went out the window. There was no evidence of chaotic behavior or strange attractors in asset returns.

• The modelers turned to neural networks—it was the early 1990s, and NNs were still fashionable. Some dissent emerged, which led to the departure of the last two junior founding partners.

• There was a surprisingly modern aspect to PC’s approach: they estimated a large number of models and averaged their predictions, with the intuition of reducing estimation variance. This happened around 1994, the same year Breiman’s bagging paper was published. Freund and Schapire’s AdaBoost paper followed the year after.

• PC decided to trade all liquid assets, all the time: FX, futures, and cash equities.

• Over time, they hired more software engineers and data experts, and fewer complexity scientists.

The book’s narrative ends in 1997 on a high note. The company was profitable. O’Connor was now part of UBS and had renewed its contract. Organizational issues were sorted out. The codebase was under control. Norman Packard and Doyne Farmer had found their roles as head of research and CEO, respectively.

To me, there are several lessons here, and one big final surprise.

The first lesson is really a confirmation of a simple belief: scientists who try to apply their pet theory to investing rarely succeed. And this happens all the time. Tom Cover’s theory was Universal Portfolios. Markowitz’s was naïve mean–variance optimization. In Farmer and Packard’s case, the application rested on a vague analogy.

However, those who are flexible enough to abandon their theory and experiment have a real chance. Investment “science” is an experimental discipline. It depends on formulating hypotheses, testing them on observational data, and refining them. Eventually, the resulting theories and models look nothing like what the researchers originally had in mind.

Jim Simons has often remarked that astronomers have done well at Renaissance Technologies. The original modeler at a prop trading firm I know was an astrophysics major from Princeton. This makes sense: astronomers are comfortable with large data sets, mathematical models, and experimental protocols for observational data. The chair of a top-five mathematics department that regularly sends PhDs to RenTech once told me that it is not the “best” graduates who join the firm. They are not the ones who did the most impressive research, but rather those with short attention spans, high curiosity, and occasional aimlessness.

In this light, the success and longevity of The Prediction Company were something of a happy accident. The founders’ strong theoretical bent could have been a hindrance, but their intellectual curiosity and flexibility saved the firm. Still, PC never took off the way RenTech—or even CFM, the French fund led by Jean-Philippe Bouchaud—did. PC had about 20 employees in 1997, and not many more in 2018. Even for such a small firm, the strategy’s capacity—the maximum achievable PnL—was insufficient to make it economically viable. By then, it had only $350M allocated by Millennium, which had taken economic control of the firm from UBS a few years earlier. In 2018, the company shut down.

Tom Bass, the author of The Predictors, was a friend of the duo. He had direct access to them during their attempts to predict roulette outcomes, which he documented in The Eudaemonic Pie. In that book, Bass never inserts himself into the story; for most of it, he is a fly on the wall. Either he was present during the conversations as they happened, or he received detailed, blow-by-blow accounts from people inside the company. You do not read a book with that level of access very often.