'Confidence' and 'Probability': Introduction

Analysts, researchers, scientists and other seekers of truth very often wish to express a level of confidence in a probabilistic judgement. For example, most people if given some time to try out a fair coin will be happy to say:

"This coin has a 50% chance of coming up heads in the next toss."

and to express 'high confidence' in such a judgement. But if asked a question about the relative ages of celebrities, we might be prepared out of total ignorance to sign up to

"There is a 50% chance that David Cameron is older than Rick Astley."

but would be considerably less happy to do so and would want to express 'low confidence' in our judgement. (Spoiler: Rick is the elder by about eight months.)

More seriously, the issue of the meaning and communication of 'confidence' is of significance in the provision of scientific and other analytical advice to policymakers. Plausibly, the lack of a coherent way of communicating confidence was one reason for the convictions of six Italian scientists and one official following the L'Aquila earthquake of 6 April 2009. Difficulties with expressing confidence bedevil the intelligence community, as expressed by Lord Butler in his review of the intelligence concerning WMDs prior to the Iraq War:

"Such assessments often include warnings that the evidence is thin (and the word ‘Judgement’ is itself a signal to the reader that it is not a statement of fact). But it is not the current JIC convention to express degrees of confidence in the judgement or to include alternative or minority hypotheses. The consequence is that the need to reach consensus may result in nuanced language. Subtleties such as “the intelligence indicates” rather than “the intelligence shows” may escape the untutored or busy reader. We also came across instances where Key Judgements unhelpfully omitted qualifications about the limitations of the intelligence which were elsewhere in the text."

Are you sure about that?  How sure?

In short, there is widely understood to be a concept of 'confidence' that needs to be attached to judgements so that researchers can express themselves clearly and readers are not misled.

However, although in my experience analysts (and their customers) have strong intuitions regarding confidence, they have a great deal of difficulty expressing what they mean by it. Those with a scientific or statistical training are apt to try to express confidence using wholly unsuitable tools such as (unfortunately-named) confidence intervals. A number of more intuitive approaches have been developed, not all of which are wholly consistent or satisfactory. The apparently-simple concept of 'confidence' turns out to be much more complex than it appears.

In the next few posts, we will explore the concept of confidence in analytical judgements, and present the results of research and survey evidence conducted by Aleph Insights. The main questions we will seek to answer are:

• Is it possible to make the concept of 'confidence' in a judgement, distinct from its probability, meaningful?
• Does our intuitive concept of 'confidence' align to one of these meaningful interpretations?
• Is it possible to design a consistent communicative tool to express 'confidence', that would be usable by analysts and comprehensible to their customers?