Overconfidence is also relatively easy to measure: it manifests itself as

*miscalibration*, which occurs when (for example) significantly more than 10% of statements assigned a 10% probability actually turn out to be true. This is easily depicted graphically, such as in Tetlock (2005):

*Tetlock (2005)*

Here, 'subjective probability' (the x-axis) is the probability actually placed by participants on the (numerous) statements in Tetlock's study. 'Objective frequency' is the observed frequency with which those statements subsequently came true. The shape of the observed calibration curves (the dotted lines) has been replicated in a number of other studies including an unpublished small-sample survey of UK government analytical product my team conducted in 2012.

Analyses of overconfidence tend to take a descriptive statistical or psychological approach. Psychologists have proposed various decompositions of the mechanisms driving overconfidence, including 'support theory', 'random support theory', 'probabilistic mental models' and others. These aim parsimoniously to identify sets of heuristics that account for observed deviations of probabilistic judgements from observed frequency.

I have found that an abstract approach, rooted in the theory of inference, and almost certainly not novel, replicates some of the observed findings using a single parameter that might stand in for a range of mechanisms. An ideal inference machine would produce a probability that is essentially a summary of the quantity of information received, expressed in terms of Shannon information content (a standard and powerful information-theoretic concept). If we apply a single modifier to information content - as though we were over- or under-estimating the quantity of information received - this produces calibration curves that are similar to the observed data:

(The family of functions is: estimated probability = p^k / (p^k + (1-p)^k), where k is an 'overconfidence parameter' modifying information content, and p is the subjective probability. This can be straightforwardly derived from the relationship between binary probability and Shannon information content.) The simplicity of this function, which has only one parameter, might make it useful for modelling the impact of miscalibration on decisionmaking, whether or not it corresponds to any real psychological mechanism.

## No comments:

Post a Comment