Thursday, 9 October 2014

Clarity on Type I and Type II Errors

The concepts of 'Type I' and 'Type II' errors are taught early on in all statistical analysis courses.  They are often referred to in intelligence contexts as well, and particularly in the field of warning, e.g. here and here.  It is important to realise that the Type I / Type II error distinction (a) is properly applied only to decisions, not to analysis, and (b) depends on an arbitrary demarcation between 'action' and 'inaction' which is difficult to support under careful examination.

A Type I error is typically described as a 'false positive'.  The archetypal example is to infer that someone has a disease, when they don't.  A Type II error is typically described as a 'false negative'.  This is where you give someone the all-clear, when in fact they have the disease.

The reason that this distinction is not applicable to analysis - assuming it's done properly - is that analysis is directed at computing a probability that something is true; e.g. the probability that your patient has cancer, or that the DPRK is about to invade South Korea.  There are all kinds of critical thinking errors you can commit to bias your probability estimate, but it doesn't make much sense to call any of them 'Type I' or 'Type II' errors.  For one thing, there is no 'positive' or 'negative' to consider, only a probability.

Type I and Type II errors don't come into the picture unless an action is involved - for example, the decision to prescribe chemotherapy or to mobilise troops.  If, when all the facts are known, you'd rather have taken a different course of action, you'd be a victim of one of these kinds of errors.  But it doesn't necessarily mean that the analysis was wrong, even though it was a component of the decision.  You could just have been unlucky.  Things which are 99% likely to be true are still false one time in a hundred.  (You'll see Type I and Type II errors in statistical analysis text-books, but this is in the context of the 'rejection' or 'non-rejection' of the 'null hypothesis' - i.e. a decision.)

The distinction between Type I and Type II errors is an arbitrary one.  There is no metaphysical distinction between 'acting' and 'not acting' (or 'acts' and 'omissions').  Prescribing nothing, or keeping troops at a low readiness level, are both actions.  The idea that they represent a 'baseline' can't be made terribly coherent in conceptual terms, and any attempt to do so is vulnerable to an arbitrary relabeling.

Imagine a metal detector which stays silent until it detects metal, when it emits a bleep.  Here, bleeping in the absence of metal would be a 'Type I error' and staying silent in the presence of metal would be a 'Type II error'.  Now imagine a 'metal-lessness detector' which continually bleeps until it detects the absence of metal, when it emits silence.  This time, the Type I and Type II errors are the other way round.  But this 'metal-lessness detector' is functionally identical to the metal detector in every respect.

Claims that Type I and Type II errors are relevant to analysis - and particularly claims that the distinction between the two is significant - should be therefore be examined critically.

No comments: