In this uncertain world, we all want to have a clearer idea of the future that lies ahead for us, our communities and our businesses. But prediction is fraught with difficulty, so there is no better time to read up on the art and science of prediction.  The Signal and the Noise deals with these challenges and is essential reading fif you want to understand the limitations of big data. The author, Nate Silver, has written extensively on the limitations of models and experts in accurately forecasting the future (and I recommend his website fivethirtyeight.com).

The Signal and the Noise lays bare the limitations of predictions and forecasts, talking about the limitations of data, the fallibilities of the human brain, and the danger of relying on automated analysis tools without an understanding of the topic being investigated as well as the data and the tools.

Nate Silver is especially clear in his belief that more data does not mean better predictions, and as the title of the book states, one of his founding principles is that any predictive model should maximize signal and minimize noise. Thus, big data does not mean better data, and as with the human mind, more information can often lead to poorer decision-making (more noise than signal).

Human instinct is to look for and find patterns in the world, even where none exist (which is the source of the Law of Prägnanz, also know as the clustering illusion, which is one of the key principle of Gestalt psychology). Although the human brain is said to be able to store 3 terabytes of information, the only way it can deal with it is by simplifying.

In the course of the book, Nate Silver discusses predictions in baseball, chess, stock markets, terrorism, weather, climate change and gambling among other topics, and references the work of Daniel Kahneman, Nassim Nicholas Taleb, Philip Tetlock, Thomas Bayes and, predictably perhaps, Sherlock Holmes. The book is relatively easy to read and packed with anecdotes and personal experiences, as the author has met and spoken to most of the people he references in the book.

He starts with the economic crash, explaining some of the key principles in the book and especially the difference between risk (where the odds are known) and uncertainty (where they are not). His discussion focuses on how many of the mistakes made in the housing bubble and financial crisis were through a lack of context in evaluating housing prices, rating agencies, financial system stability and history of recession events (in his words, they were evaluating based on too limited a sample of information).

He goes on to discuss that this is not just a failing of you and me, but a failing of experts. Indeed, much work (like that of Philip Tetlock) shows that experts often make worse predictions than others, most especially when they are focused on one domain of expertise (the hedgehog) rather than a general knowledge across domains (the fox). While hedgehogs are usually more confident in their predictions (and are therefore the ones who get invited to be TV pundits), they actually make worse predictions than their more cautious and circumspect fox counterparts, who actually make slightly better predictions because they incorporate ideas from multiple disciplines into their thinking.

Above all, the book is an argument for greater probabilistic thinking in business and life (hence the prominence of Thomas Bayes). In the spirit of Bayes, Nate Silver also argues that predictions should be constantly updated to take account of new information, and should be based on a consensus view through the merging of different opinions and data sets (i.e., aggregating knowledge).

Weather forecasts are one of the success stories of the book (unlike economics), where Nate Silver believes that standards have improved and that methods are based on the approaches he recommends for making better (but not perfect) probabilistic predictions.

In a world where the volume of information is increasing exponentially, there is more and more information that has very little or no value in making useful predictions. As Nate Silver states, the trick is to know the difference between what we know and what we think we know, and to use the tools of predictive analytics to try and close the gap, which requires a leap into Bayesian thinking.

I enjoyed reading The Signal and the Noise. Although I would have liked to see a little more practical advice on dealing with the information age, the book is packed full of wisdom and a clear view of where analytics can help, and where it can’t. It should be essential reading for anyone who wants to create value from data and information.

[This is a lightly edited version of a book review originally written in 2015.]