Noise in Judgement

'Noise: A Flaw in Human Judgement' by Daniel Kahneman, Olivier Sibony, and Cass Sunstein is one of my top recommended readings. While I'm still very much a junior behavioural scientist, as I read I found myself connecting the insights to my experiences within my own domains of expertise.

The authors make use of Pythagorean theorem and triangles to explain the relationship between different components of error within human judgement. But what I liked most were the visualisations.

I've recreated the key diagram as an interactive infographic. Clicking on a component will render a description below the diagram:

Mean Squared Error (MSE) Mean Squared Error (MSE) Mean Squared Error (MSE) Bias² Bias² System Noise² System Noise² Level Noise² Level Noise² Pattern Noise² Pattern Noise² Occasion Noise² OccasionNoise² Stable Pattern Noise² StablePattern Noise²

Level Noise²

Level noise is the result of consistent deviations from the average, of individual judges, in a particular direction. If you calculate the mean judgement of cases for each judge then plot those means —a scatter plot works well— the resultant variation is level noise.

At a shallow glance this looks just like bias, but it's not. Bias is the mean deviation from an expected or baseline value. This is noise because it's looking at the variation of individual deviations and not the aggregation of those individual variations; there is no baseline here.

The example given in 'Noise: A Flaw in Human Judgement' is the difference in court judges desposition and sentencing. Some judges are strict and some are lenient, giving harsher and softer penalties respectively.

Fitness to a purpose evaluations, such as job interviewing, are also good examples. Individual interviewers differ somewhat in their judgements of the candidates. This can cause issues if they vary widely in comparison to the mean of the panel. The overly strict members may reject everyone as unsuitable while the highly compassionate judge all as worthy. Considering the objective is to come to a consensus about who to hire, the decision making process can become tedious and painful for those involved.

Such cases often end with an unstructured and error ridden methodology being applied. Those who are most vocal, most charming, or the HIghest Paid Persons Opinion (HIPPO) tend to stongly influence the results. Which is the exact opposite to the purposes of the panel!

Sources

[book] Noise: A Flaw in Human Judgement. 2021. D Kahneman, O Sibony, C R Sunstein [Pub] Harper Collins [ISBN] 978-0-00-830903-9.

[book] Knowledge and decisions. 1996. T Sowell [Pub] Basic Books [ISBN] 978-0465037384.