Saturday, 20 December 2014

Mandy Rice-Davies and the Non-diagnostic Evidence

Mandy Rice-Davies, who died yesterday aged 71, is known primarily for her pithy contribution to popular information theory literature.  When told in court that Lord Astor had denied having an affair with her, she replied simply: "He would, wouldn't he?"

The point is that if the conditional likelihood of any piece of evidence (Lord Astor's denial of having met Mandy) is the same (100%) under a given hypothesis (Lord Astor met Mandy) as under its negation (Lord Astor did not meet Mandy), this evidence will be non-diagnostic with respect to the hypothesis, as a straightforward implication of Bayes' Theorem.  Mandy Rice-Davies' formulation is of course superior in brevity.

Friday, 19 December 2014

Sony Hacker Threats; Risk Assessment Difficulties

In response to threats from possibly-state-led hackers, Sony has pulled 'The Interview' from cinemas, almost certainly guaranteeing a significantly higher viewership when it proliferates on the internet in leaked form.  It's the latest in a surprising sequence of events, with more no doubt to come.

The debate about the wisdom of this decision has naturally focused on the identity of the hackers, the ethics of responding to coercion, and the impact on free speech.  But what if we focus purely on the risk assessment element of Sony's decision?  Can we quantify the risk to public safety that Sony was hoping to mitigate?  Trying to do so exposes some of the more interesting and difficult elements to risk assessment when we have relatively-unprecedented developments.

The background risk from terrorist attacks to entertainment events is very low. According to the Global Terrorism Database (GTD), there are only around 18 attacks worldwide per year on targets in the 'entertainment / cultural / stadiums / casinos' category, with an average of fewer than 1 of these in the US.  Attacks in this category are slightly less deadly than average, with a mean of 1.7 deaths per attack.  

The US deaths-per-attack figure is significantly lower, although curiously the GTD doesn't include the 2012 Aurora cinema attack in Colorado (probably for definitional reasons).  Even including this attack, though, the average risk of death from terrorism in cinemas for a US citizen is around 1 in 500,000,000 per year (one death, on average, US-wide, every couple of years or so).  The average US citizen goes to the cinema about four times a year, so a trip to the cinema exposes an average American to about a 1 in 2,000,000,000 chance of death from terrorism.  That's about 100 times lower than the risk of dying in a five-mile round trip by car to get to the cinema in the first place.

Photo: Fernando de Sousa
A generally-safe place to be

That's under normal circumstances.  What about when a major film distributor has faced explicit, coercive threats from capable hackers that might or might not be backed by nuclear-armed states?  This is when things get difficult.  Robust approaches to risk assessment, particularly for low-probability events, usually start by building a reference-class - a set of relevantly-similar events that can be used to form statistical base-rates to which scenario-specific modifiers can be applied.  In this case, the reference-class is so small (approaching 0) that the specifics, and assumptions about them, dominate the estimate.

The hackers threatened attacks on a 9/11 scale if the film were screened.  If these threats were absolutely credible, the expected number of deaths on the film's opening weekend would be in the hundreds or possibly thousands.  If the threats are entirely empty, then the expected number of deaths from terrorism in the opening weekend would be more like the background level of around one one-hundredth of a death.

Whether the risk is at the background level, or whether it is at the "hackers' threat" level, depends on what intelligence you have and which assumptions you make.  In between are five orders-of-magnitude of uncertainty in terms of expected impact.  Did Sony make the right call?  Judging by reviews of advance screenings and even Sony executives themselves, the answer might be 'yes' whatever you think the risk was.  

Saturday, 13 December 2014

Peripheral Vision in Animals and Organisations

Vision-related tasks in animals are accomplished in an extraordinary range of ways. There are at least ten basic eye structures, each of which can be adapted along dimensions such as colour perception, field-of-view, photoreceptor density, field of focus and so on. Examining what an animal's eyes seem to be optimised for can lead to insights about the resource, threat and physical environment they face.
Apex predators have forward-facing eyes, while prey animals tend to have eyes either side of their heads to maximise field of vision. Horizon-scanning animals such as lions and horses have a horizontal visual 'streak' of high photoreceptor density, while birds and tree-dwelling animals, including primates, tend to have a broadly circular central 'fovea' where visual acuity is highest. Colour perception tends to be much more discriminatory among land animals than in those living in the sea, where the chromatic spectrum is significantly narrower due to absorption by water.

In humans, the make-up of our foveal and peripheral vision systems are quite distinct, suggesting that they are optimised to different tasks. Foveal vision is extremely narrow, very high-resolution, and dominated by colour-sensitive 'cone' cells that require high levels of light to function. In contrast, peripheral vision is accomplished with 'rod' cells that respond well in low light levels but lack colour discrimination.  

Rods and cones by angle from fovea (source: Wikipedia)

However, we are usually unaware of quite how narrow the angle of our high-acuity vision is. Only when we experience visual disturbances such as scintillating scotoma do we realise how difficult it is to perceive detail (e.g. for reading) outside of this tiny 'spotlight'. Most of our perception of the world is in any case the result of visual processing - thought to account for around a quarter of the brain's structure - and so our conscious acquaintance with the information our eyes are receiving is somewhat indirect. But we are primarily unaware of the fovea's narrowness, of course, because our head and eyes are highly mobile. Wherever we look, we see a detailed picture, and by-and-large the things we want to look at only rarely move faster than our eyes can catch. A foveal level of vision everywhere would be unnecessary as well as sacrificing perception in low light.

Instead of giving us high detail everywhere, human vision represents an efficient partnership between a small, mobile, highly-detailed foveal component, and a wide, low-resolution peripheral component. Peripheral vision's vital role is to identify objects of possible interest in order to cue examination by the fovea. Without peripheral vision to support it, the fovea would be completely debilitated, as the experience of tunnel vision sufferers attests.

Many organisations with analytical components have re-invented this division of responsibility, which aligns broadly to a distinction between hypothesis-generating and hypothesis-testing analytical tasks. Organisational peripheral vision involves identifying novel threats and opportunities, seeking new markets, and identifying new products and productive technology. Organisational foveal vision involves gathering data on known threats and opportunities, analysing existing markets, and exploiting existing products and technology.  Often, but not always, these tasks are conducted by different parts of the organisation.

The importance of peripheral vision is easy to underestimate though. By necessity, it does its job out of the spotlight. When analytical budgets get cut, it can look more attractive to cut horizon-scanning, threat-driven functions than to cut those functions dedicated to exploiting known profit drivers. But organisational tunnel vision carries existential risks for many organisations. Failure to spot emerging threats has arguably been a significant driver behind a host of disastrous business decisions, including General Motors' failure to adapt to the growing market for smaller cars, Blockbuster failing to acquire Netflix, Kodak's tardiness in promoting digital products, and Excite's refusal to buy Google in 1999.

The consequences for warning failure in defence and security can of course be more significant. A focus on known threats at the expense of novel ones may have lain behind failures adequately to anticipate and respond to developments such as the stationing of Soviet ICBMs in Cuba, the Iranian Revolution, or the September 11 attacks.

Although our everyday perceptual experience may relegate it to a supporting role, peripheral vision is essential for our survival, as an animal or an organisation.

Wednesday, 10 December 2014

Affine Transformations of Payoffs should not affect Decision-making

Behavioural economics has unearthed interesting features of decision-making that sometimes seem to violate rationality assumptions.  These include loss aversion - people's willingness to take risks to avoid losses, but to avoid risks to take certain gains - and the endowment effect, in which people put a higher price on assets they own compared to those they don't.

These findings don't necessarily violate rationality.  They might also support the hypothesis that people have complex objectives comprising many different factors.  For instance, if losing confers psychological or social costs in itself, additional to any material loss, taking risks to avoid losses might be optimal.

But where objectives are well understood - say in business, where (mostly) the objective of a firm is to maximise long-run profits - optimal decision-making won't be influenced by anything other than the outcomes associated with each possible course of action, their relative probabilities, and the organisation's appetite for risk.

One possibly-surprising feature of optimal decisions is that they are robust to affine transformations of payoffs.  What this means is that multiplying all payoffs by the same (positive) factor, or adding or subtracting a constant sum from all payoffs, should make no difference to the optimal decision.  This means that affine transformations make a good test of a proposed decision rule.  If all the outcomes were halved, or doubled, or all worth exactly £1m more than they are, the decision rule should always identify the same choice.  Inter alia, this is one reason that lump sums (like licensing fees or the poll tax) are considered to be the least distortionary types of tax.

Evidential Support is More Complex than 'For' or 'Against'

A naive view of probability pictures it as the result of a sort of tug-of-war between arguments for a statement and arguments against that statement.  If the arguments in favour of a proposition beat the arguments against, the probability will be high.  If it's the other way round, it'll be low.  Some critical thinking techniques, such as argument mapping or analysis of competing hypotheses, can promote this way of thinking, which is unfortunate because it's misleading.

When evaluating a hypothesis or scenario, the evidence cannot easily be split into 'for' and 'against' arguments for separate consideration.  Instead, all of the evidence needs to be considered as a whole.  The final probability of the target hypothesis is determined by the likelihood of all of the evidence conditioned under both that hypothesis and its alternatives, and is not straightforwardly a summation of parts of the evidence.

As an example, take these three statements:

A: "Jones lives in Eton College"
B: "Jones is more than twenty years old"
C: "Jones is female"

Taken alone, (A) is certainly evidence against (B).  Knowing that Jones is at Eton makes it highly likely that he's a boy or a young man.  But if you know (C), that Jones is female, (A) becomes very strong evidence in favour of (B), since it's highly likely she's a teacher or the wife of one.  

Items of evidence cannot, in other words, be treated in isolation from one another.  

Friday, 5 December 2014

Evidence of Absence

How much absence of evidence do you need for it to constitute evidence of absence?  Despite what the maxim says, all absence of evidence is evidence of absence, assuming that the evidence in question had a chance to show itself but didn't.

The question of when to conclude that something is absent is a surprisingly common problem.  When you are searching a suspect's house, when do you decide to give up looking?  When you are waiting for a lift, when do you conclude the lift is broken?  When you are searching satellite imagery for nuclear facilities, when can you assume they're not there?  When you are looking for ultrasound 'evidence' that a foetus is a boy, when do you conclude it's a girl?

These possible-needle-in-haystack problems (ones where we're not sure the needle is actually there) are all governed by the same underlying information process: in any given period of time, you have some probability of establishing a hypothesis with 100% certainty, but no chance per se of absolutely falsifying it.

It's in there somewhere... maybe

We can put numbers on the question by considering two search methods which represent extreme cases for favourable searching.  The first is when you exhaustively search the haystack, straw by straw, until the needle is found, in very much the way that the Mythbusters did.  The second is when you randomly choose bits of the haystack to search, so that at any given time you might be searching part of the haystack you've already ruled out.

The first process is governed by a simple likelihood function.  Assume it takes a hundred man-hours to search a large haystack for a possible needle.  The probability of not finding it if it's there is (100-t)%, where t is the search time so far.  (The probability of not finding it if it's not there is of course always 100%).  The second process - random searching with replacement - is governed by an exponential function, which is something that appears regularly in dynamic information problems.  Assuming the same search rate above, the probability of not finding the needle after t hours is (100-100e^(0.01t))%, where e is the natural logarithm 2.71828...

In the first process, you have a 50% chance of finding the needle, if it's there, after half the time.  In the second process you have a 50% chance of finding it after around 70% of the time.  If, when we start, we believe there's a 50% chance that the needle's in there at all, the graph of the probability it's there, over time, looks like this:

So intuition, which suggests systematic searching is better than random searching, is right in this case.  It is possible to imagine search strategies which are worse than random searching with replacement, which would involve being more likely to search areas you'd already looked at.  But in general, if it would take you time T to search the whole area, absence of evidence after time T will mean that absence is at least three times more likely, relative to presence, than it was when you started.