"This coin has a 50% chance of coming up heads in the next toss."
and to express 'high confidence' in such a judgement. But if asked a question about the relative ages of celebrities, we might be prepared out of total ignorance to sign up to
"There is a 50% chance that David Cameron is older than Rick Astley."
but would be considerably less happy to do so and would want to express 'low confidence' in our judgement. (Spoiler: Rick is the elder by about eight months.)
"Such assessments often include warnings that the evidence is thin (and the word ‘Judgement’ is itself a signal to the reader that it is not a statement of fact). But it is not the current JIC convention to express degrees of confidence in the judgement or to include alternative or minority hypotheses. The consequence is that the need to reach consensus may result in nuanced language. Subtleties such as “the intelligence indicates” rather than “the intelligence shows” may escape the untutored or busy reader. We also came across instances where Key Judgements unhelpfully omitted qualifications about the limitations of the intelligence which were elsewhere in the text."
|Are you sure about that? How sure?|
However, although in my experience analysts (and their customers) have strong intuitions regarding confidence, they have a great deal of difficulty expressing what they mean by it. Those with a scientific or statistical training are apt to try to express confidence using wholly unsuitable tools such as (unfortunately-named) confidence intervals. A number of more intuitive approaches have been developed, not all of which are wholly consistent or satisfactory. The apparently-simple concept of 'confidence' turns out to be much more complex than it appears.
In the next few posts, we will explore the concept of confidence in analytical judgements, and present the results of research and survey evidence conducted by Aleph Insights. The main questions we will seek to answer are:
- Is it possible to make the concept of 'confidence' in a judgement, distinct from its probability, meaningful?
- Does our intuitive concept of 'confidence' align to one of these meaningful interpretations?
- Is it possible to design a consistent communicative tool to express 'confidence', that would be usable by analysts and comprehensible to their customers?