Monday, 20 October 2014

Logarithmic Probability Scores

Expressing probability as a percentage is something we're all used to doing, but in many ways it's of limited usefulness.  It's particularly inadequate for communicating small probabilities of the kinds considered in risk assessments such as the UK's National Security Risk Assessment which looks at a range of risks whose probabilities might vary in orders of magnitude, from say one in a million to one in ten.  For these kinds of situations - where we're more interested in the order of magnitude than in the precise percentage - a logarithmic ('log') scale might be a better communication tool, and potentially more likely to support optimal decisions about risk.

On a log scale, a one point increase equates to an increase in magnitude by a constant factor, usually ten (since it makes the maths easier).  A log scale might start at '0' for a 1 in 1,000,000 probability, move to '1' for 1 in 100,000, and so on through to '5' for 1 in 10 (i.e. 10%) and '6' for 1 in 1 (i.e. 100% probable).  (A more mathematically-purist probability scale would actually top out at '0' for 100%, and use increasingly negative numbers for ever-lower probabilities.)  A log scale also brings the advantage that, if paired with a log scale for impact, expected impact - which is a highly decision-relevant metric - can be calculated by simply adding the scores up (since it's equivalent to multiplying together a probability and an impact).  One thing it can't do, though, is express a 0% probability (although arguably nothing should ever be assigned a 0% probability unless it's logically impossible).

For these reasons, log scales are used to simplify communications in a number of risk-related domains where the objects of interest vary in order-of-magnitude.  The Richter Scale is a log scale.  The Torino scale is used to combine the impact probability and kinetic energy for near-earth objects:

Several intelligence analysis organisations have developed standardised lexicons for expressing probabilities, such as the UK Defence Intelligence's uncertainty yardstick (last slide) and the US National Intelligence Council's 'What we Mean When We Say' (p.5) guidance.  To my knowledge, however, there are no standardised log scales used in these areas.  There may be an argument for their development, to enable easier communication and comparison of risk judgements concerning high-impact, low-probability events across the intelligence community and government more widely.  

No comments: