Friday, 19 December 2014

Sony Hacker Threats; Risk Assessment Difficulties

In response to threats from possibly-state-led hackers, Sony has pulled 'The Interview' from cinemas, almost certainly guaranteeing a significantly higher viewership when it proliferates on the internet in leaked form.  It's the latest in a surprising sequence of events, with more no doubt to come.

The debate about the wisdom of this decision has naturally focused on the identity of the hackers, the ethics of responding to coercion, and the impact on free speech.  But what if we focus purely on the risk assessment element of Sony's decision?  Can we quantify the risk to public safety that Sony was hoping to mitigate?  Trying to do so exposes some of the more interesting and difficult elements to risk assessment when we have relatively-unprecedented developments.

The background risk from terrorist attacks to entertainment events is very low. According to the Global Terrorism Database (GTD), there are only around 18 attacks worldwide per year on targets in the 'entertainment / cultural / stadiums / casinos' category, with an average of fewer than 1 of these in the US.  Attacks in this category are slightly less deadly than average, with a mean of 1.7 deaths per attack.  

The US deaths-per-attack figure is significantly lower, although curiously the GTD doesn't include the 2012 Aurora cinema attack in Colorado (probably for definitional reasons).  Even including this attack, though, the average risk of death from terrorism in cinemas for a US citizen is around 1 in 500,000,000 per year (one death, on average, US-wide, every couple of years or so).  The average US citizen goes to the cinema about four times a year, so a trip to the cinema exposes an average American to about a 1 in 2,000,000,000 chance of death from terrorism.  That's about 100 times lower than the risk of dying in a five-mile round trip by car to get to the cinema in the first place.

Photo: Fernando de Sousa
A generally-safe place to be

That's under normal circumstances.  What about when a major film distributor has faced explicit, coercive threats from capable hackers that might or might not be backed by nuclear-armed states?  This is when things get difficult.  Robust approaches to risk assessment, particularly for low-probability events, usually start by building a reference-class - a set of relevantly-similar events that can be used to form statistical base-rates to which scenario-specific modifiers can be applied.  In this case, the reference-class is so small (approaching 0) that the specifics, and assumptions about them, dominate the estimate.

The hackers threatened attacks on a 9/11 scale if the film were screened.  If these threats were absolutely credible, the expected number of deaths on the film's opening weekend would be in the hundreds or possibly thousands.  If the threats are entirely empty, then the expected number of deaths from terrorism in the opening weekend would be more like the background level of around one one-hundredth of a death.

Whether the risk is at the background level, or whether it is at the "hackers' threat" level, depends on what intelligence you have and which assumptions you make.  In between are five orders-of-magnitude of uncertainty in terms of expected impact.  Did Sony make the right call?  Judging by reviews of advance screenings and even Sony executives themselves, the answer might be 'yes' whatever you think the risk was.  

No comments: