Noise in Judgement

'Noise: A Flaw in Human Judgement' by Daniel Kahneman, Olivier Sibony, and Cass Sunstein is one of my top recommended readings. While I'm still very much a junior behavioural scientist, as I read I found myself connecting the insights to my experiences within my own domains of expertise.

The authors make use of Pythagorean theorem and triangles to explain the relationship between different components of error within human judgement. But what I liked most were the visualisations.

I've recreated the key diagram as an interactive infographic. Clicking on a component will render a description below the diagram:

Mean Squared Error (MSE) Mean Squared Error (MSE) Mean Squared Error (MSE) Bias² Bias² System Noise² System Noise² Level Noise² Level Noise² Pattern Noise² Pattern Noise² Occasion Noise² OccasionNoise² Stable Pattern Noise² StablePattern Noise²

Mean Squared Error (MSE)

MSE measures the average squared difference between a set of estimated values and actual values.

In the context of error in judgement, it is a measure of error including both noise, how spread out the judgements are, and bias, how much they deviate from a baseline. This means a reduction in either noise or bias will reduce error in judgement; at least on average.

This is important because we habitually focus on bias when discussing and managing error. But noise may play an equal or greater role. This is easier to see when employing Pythagorean theorem. We do this by taking the generic equation (a² = b² + c²), injecting the measures of error (MSE = noise² + bias²), then plotting a graphic such as the one above. I've also added the types of noise also mentioned in Noise.

Consider the graphic as a visual reference. If we reduce the size of the noise component while fixing bias then MSE must decrease. The antithesis is also true. Increasing noise not only increases overall error but also increases the share of the error caused by noise. That is, bias becomes less and less significant as noise increases, and vice versa.

When it comes to data analysis and judgement, the first thing we should consider when evaluating potential error is the ratio of noise to bias. If noise turns out to be the larger component then it is noise reduction, not bias reduction, that is likely to give the greatest value as a first step.

Furthermore, unwanted variation can interfere with our ability to effectively visualise and manage bias. That is, the more we reduce noise the clearer picture we get of bias and the better placed we are to deal with it.

Sources

[book] Noise: A Flaw in Human Judgement. 2021. D Kahneman, O Sibony, C R Sunstein [Pub] Harper Collins [ISBN] 978-0-00-830903-9.

[book] Knowledge and decisions. 1996. T Sowell [Pub] Basic Books [ISBN] 978-0465037384.