Saturday, November 26, 2016

Physics of Wall Street: A Brief History of Predicting the Unpredictable

With The Name of God, The Most Merciful. The Most Beneficent,
And The Excellent Helper, And The Excellent Protector
(As per my own research crashes could be predicted 6 to 12 months in advance. Holds true for all the crashes in Indian Equity Markets. I have recorded the same phenomenon in Gold.)

Following is review of the book by a certified Vine Amazon Customer:

It is not easy to come up with star ratings for this book. I will split the difference between two and four stars. Here is why...

On one hand, I consider it to be a chronology of scientific efforts to predict markets beginning as early as the 18th Century, all the way up to 2012. As such, it is very interesting in terms of some the best known historical names dabbling in it. The storytelling in the book is not very good, but despite that it kept my interest through many chapters.

On the other hand, I consider this book to be an attempt to explain (to the layman) how science can be used to predict markets. To that end, the examples of Simons and Sornette (and their spectacular success on Wall Street) are presented without going into details. One cannot justify everything by the brilliance of these men alone. If they indeed were successful based on their knowledge of physics, how they managed to do that should have been analyzed in the book. Instead, the author takes a very long-winded tour of random statistical distributions starting with Gaussians, then he moves on to Cauchy distributions and other fat-tailed distributions, and how they may be relevant to markets. If that was all there is for the scientific methodology of market prediction, you would not need physicists like Simons and Sornette. Anybody with some basic math and statistics will do fine. The physicists do far more than that, and none of that was discussed in the book. They come up with models of dynamic market balance, they convert these models to differential equations to be solved, and they (approximately) solve them (on fast super-computers.) It would have been fascinating if the author had any details available on that subject.

On the other hand, this book discusses a fascinating question on and off many times: can markets really be predicted? All indications are that markets can be predicted by sophisticated mathematical models (done by physicists) most of the time (when markets behave normally.) But there are times when markets do something really wild. The market crashes of 1929, 1989, 1997, and 2007 are examples. The real question is this: can a market crash be predicted a year in advance? The author discusses this topic too and tries to explain it as reaching the critical point when some rapid and discontinuous change takes place. The physics is very rich in well understood critical phenomena, many of which can be applied to markets. That's where the non-linear dynamics comes in. I believe (I don't know this for a fact) that some of the physicists on Wall Street can predict a major market crash several days (or even possibly weeks) ahead. But they can't do that a year in advance. The non-linear dynamics is notoriously sensitive to initial conditions, and the quality of the prediction deteriorates as we project farther and farther out into the future.

On the other hand, this book never discusses in detail one of the major dilemmas of market prediction. We (physicists) take great delight in discovering the laws of nature so that we can predict how it behaves. In this endeavor, we tacitly assume that nature continues to behave in the same manner before and after our discovery. The fact that we suddenly know how it behaves does not change nature's behavior. This tacit assumption almost certainly fails with markets if the discovery is made public. There are a hoard of economists out there who would trample on each other for a better predictive model of markets, which may earn them the Nobel Prize. If indeed such a model is ever developed that can predict the markets well (not only in normal times but also shortly before market crashes) it will spell its own doom. In response to this new model, the behavior of the markets will change sufficiently to counter the prediction and render it useless. The only way such a model remains successful is by not making it public, and guarding it as a trade secret. That's how the likes of Simons and Sornette succeed, while most others eventually fail. Simons and Sornette never reveal how exactly they predict markets using physics. The author could have written a chapter (or several chapters) about this, but he did not.

Finally, the worst part of this book is about the inaccuracies it contains about who did what. Most of these are physics related (quite strangely since the author is supposed to know that), and not about the main topic of the book. For that reason, maybe they are forgivable. For example, the author implies that the gauge theory provides us a way of translating vectors from one point of curved space-time to another. While we physicists know how that should be done, translating points in curved space-time has very little to do with gauge theories. The author later corrects himself by acknowledging that the gauge theories were something else altogether and correctly credits Yang and Mills for discovering them in 1953. Never mentioned in the book is the fact that the first gauge theory even predates Yang and Mills, all the way back to 1879. It is none other than Maxwell's classical electrodynamics, even though Maxwell himself never used that term.

So in summary, if you know very little about quantitative market prediction, I highly recommend the book, you will learn things from this book: four stars. If you have some knowledge about it, then you will not get much out of the book except a few names and places in history, not all of which are correct: two stars. I will average the two ratings.