Friday, 27 May 2016

Skill and Luck

The extent to which any outcome has been determined by skill or luck is a template for debate that will probably always be in demand. Ecclesiastes 9:11 says that "chance happeneth to them all". Seneca's Agamemnon describes the influence of fortune on world affairs: "Never have sceptres obtained calm peace or certain tenure; care on care weighs them down, and ever do fresh storms vex their souls". In probably the most famous lines ever written, Hamlet wonders about the possibility of overcoming the "slings and arrows of outrageous fortune".

Fortune favours the lucky

More recently, the birth of probability theory was driven by a desire to understand outcomes in games of chance, so that profit-making (or loss-minimising) choices could be made. Even the law needs a way of deciding whether something is a 'game of skill', for the purposes (for example) of applying VAT or restricting gambling: section 14(5) of the 2005 Gambling Act defines it in the following terms:
A process which requires persons to exercise skill or judgment or to display knowledge shall be treated for the purposes of this section as relying wholly on chance if - (a) the requirement cannot reasonably be expected to prevent a significant proportion of persons who participate in the arrangement of which the process forms part from receiving a prize, and (b) the requirement cannot reasonably be expected to prevent a significant proportion of persons who wish to participate in that arrangement from doing so.
Like most law, of course, this is open to interpretation, particularly surrounding 'significant'. It's this legislation, incidentally, which motivates the seemingly bizarre practice of linking phone-in competitions to absurdly easy multiple-choice questions, the theory being that as long as a 'significant' proportion of people won't know the answer, it's officially a game of skill.

The psychology of skill perception and the effect of randomness on outcomes throws up interesting findings. There are a number of 'self-serving biases' that mean we are more likely to attribute our successes to skill, and our failures to chance. Animal studies suggest that compulsive behaviour will emerge more strongly when rewards are a combination of effort and luck, than when they are directly linked only to effort.

In popular culture, the issue of luck versus skill is a live topic in game design, both board and electronic. Higher randomness - more 'luck' - leads to more-even outcomes and potentially more-dramatic in-game events, but has a cost in terms of rewarding effort and in making it harder for designers to control emergent narratives. It's a tricky trade-off. Is it real? Do we have to make a trade-off, or can we somehow combine the best of 'luck' with the best of 'skill'? What, actually, are 'luck' and 'skill'? This is not a straightforward question.

As an analysis, the legal approach is perhaps the most unsophisticated with its supposition of a hard distinction between skill games and gambling games. No-one really believes this, necessary though the pretence is for legal purposes. But it does have the advantage of being based in outcomes rather than game characteristics: a game is a game of skill if significant numbers of people can't expect to win. This works quite well.

At the other extreme is the view put forward here by Magic: The Gathering designer Richard Garfield, which proposes that 'luck' and 'skill' are orthogonal categories, and which implies among other things that situations can be determined by lots of both, or neither. Garfield doesn't really analyse what those things mean though, and relies on our intuition. But from the context it seems he sees 'skill' as something like 'the amount of effort you can invest to improve your chances of winning' and 'luck' as something like the complement of the probability of winning, if you've invested everything you can in it. These are orthogonal, but the definition of 'skill' is hard to operationalise without circularity - if it isn't somehow related to 'probability of winning', it's hard to see how it can be considered 'skill' in the context of the game.

An alternative approach might be to think of the 'skill content' of a game in terms of the highest probability of winning that it's possible to have ex ante, assuming an opponent who consistently plays as badly as possible. If the probability of winning in this extreme case were 100% (as it would be, for instance, for noughts-and-crosses or chess) then the game is entirely skill-based; if it were 50% (as it would be, for instance, for beggar-my-neighbour) then the game is entirely chance-based. A game where the highest win probability were 75% might be said on this basis to be determined by skill and luck in equal measure.

The problem with this definition, however, is that in practice nearly all real-life games are trivially easy to lose at, even paradigmatically luck-heavy ones such as Yahtzee or Cribbage. This means that nearly all games will come out as highly skill-driven by the above definition. On one hand, perhaps this indicates that we're not aware of quite how many decisions we need to make even in relatively-simple games. Or it might mean that most of the decisions in a game like Cribbage are easy ones that relatively-new players are unlikely to get wrong, so the 'consistently bad player' is never, in practice, encountered as an opponent.

I totally meant to do that

What this suggests is that a definition based on the optimal decision-path in a game is unlikely to capture our intuitions about what it means for a game to depend on skill or luck. What also seems important is the impact of sub-optimal play. A definition of 'skill' taking this into account might refer to the probability of a player making the optimal decision in any given circumstance, or to some measure of how near to optimal their average decision was. With a definition of skill along these lines, it seems intuitive that a heavily skill-driven game would give disproportionately high rewards to small differences in the probability of making an optimal choice.

A definition along these lines would also allow us to respect the fact that some games might be skill-driven for high levels of skill, but luck-driven at low levels. Beginners in a game like Settlers of Catan will probably feel that the game is driven by the dice rolls. More-advanced players - those more likely to make near-optimal choices - will instead notice the impact of decisions, and advanced players will consistently beat amateur players, though not with the predictability of, say, chess. A definition of 'skill' based on proximity to optimal play might also help provide explanatory power for the outcomes of games with different decision profiles: for example, all things being equal, games with lots of decisions ought to reward skill more than those with fewer, more-significant ones, where even a bad player might just get lucky.

Another approach might be to build on the insight that 'luck' is, in fact, nothing more than a lack of information: how much information do we have about who is going to win, before a game starts? An operational definition of 'skill' using this approach might be based on the extent to which knowledge about players (e.g. their past performance against other players) would enable us to predict the outcome. So the more predictive player rankings are, the higher the extent to which we can assume skill drives the outcome. This definition has the advantage of not requiring any knowledge of the game itself, circumventing computational problems with identifying the optimal outcome.

So the question of what 'luck' and 'skill' are in the context of determining the outcome of games is not straightforward. A separate question is what the optimal balance of these things are, if we want to make a game fun. Here, no doubt, are matters of taste. Higher luck certainly benefits amateur players in terms of their chances of winning, but lowers the rewards associated with getting better. People's motivation in playing games differ. One thing that might be important is the story that emerges when playing a game, reflected in one side's probability of winning over time. A game that was entirely unpredictable until the end would be unsatisfying, but one which appointed a winner early on would be tedious. Perhaps the ideal game produces something akin to the three-act dramatic structure?

Lucky or skilful?
'Games', of course, includes far more than just games. We could think about this problem in the context of many real-life situations. To what extent is the outcome of an election luck- or skill-driven? What about a war? Or one's achievements over a lifetime? If we struggle with defining 'luck' and 'skill' in abstract games, these questions will not be easy to answer, but that doesn't make them less meaningful or profound.

Podcast: Hamburgers

Peter, Fraser and Nick discuss the threat posed by rare hamburgers, and ways of assessing risk in the absence of data. To subscribe to the podcast, add this RSS feed to your preferred player.

Friday, 20 May 2016

Podcast: Celebrity Wave of Death

Peter, Nick and Fraser discuss the 2016 spike in celebrity deaths, how the scientific method applies to analysing it, and some of the fundamental constraints of fame. To subscribe to the podcast, add this RSS feed to your preferred player.

Monday, 16 May 2016

Podcast: Cruftsgate

Nick, Peter and Fraser discuss what Cruftsgate tells us about classification and categorisation. To subscribe to the podcast, add this RSS feed to your preferred player.

Friday, 13 May 2016


Governments, organisations, and individuals employ a range of linguistic deception strategies. These include silence, denial, PR, propaganda, spin and outright lying. Verbal (including written) communication is an extremely dense medium for information and it makes sense for deceivers to invest in wordsmithing. The aim of deception is to induce a decision error on someone else's part in a way that advantages you and disadvantages them. What it involves in practice is interference with the opponent's information-set so that the decision you want them to take appears, to them, optimal.

Knowing how to do this in practice is hard, not easily reduced to general principles, and involves a lot of empirical knowledge about the systems in which you're operating. Successful deception in a game like poker depends on knowing how 'weak' looks when you're strong, or vice versa. Deceiving shareholders into thinking you're profitable instead of loss-making depends on knowing what profitable firms look like, and pushing out signals that resemble them and hiding ones that don't. Convincing enemy governments that your intentions are other than they are can involve extremely subtle and complex activity that is itself designed to resemble deception about something else.

The 1989 Hillsborough disaster, in the news again in April following the verdict at an inquest that the victims were unlawfully killed, gives us a practical example of organisational deception. In the aftermath of the disaster, police statements made on or close to the day were altered - 'sanitised' - before being made public, a fact only unearthed by the Hillsborough Independent Panel more than two decades later. Many of the alterations seem clearly designed to absolve the West Yorkshire Police of whatever responsibility they may have felt they bore.

Examples of this 'sanitisation' (among many) can be found in the changes made to the report of PC Martin McLoughlin. These include altering "...I was stood with PS 266 Jacques, the only officer in our serial who had a personal radio..." to "...I was stood with PS 266 Jacques, who had a personal radio...", and "I heard an officer asking desperately for a gate to be opened" to "I heard an officer asking for a gate to be opened".

This 'sanitisation' is different from redaction. Redaction works if there is such a wide range of possibilities for the redacted words (names and addresses make ideal redactees) that it doesn't much matter that people know where the gaps are. It's a very honest kind of deception. The sanitisation of the police statements widens the reader's uncertainty more subtly. The publicly-released statements were not untrue, but their information content has been reduced, widening the range of possibility to include (for example) that there might have been many officers with radios, or that the request to open the gate was a rather routine one.

A known unknown
This is considerably more deceptive than redaction because it conceals the deception itself and can therefore rely on linguistic implicatures: "if PS 266 Jacques had been the only officer with a radio, surely PC McLoughlin would have told us, so we have no reason to think he was..." If we're looking for cover-ups, we shouldn't be looking for redacted documents. Real cover-ups - good ones - don't leave a trace.

Friday, 6 May 2016

Podcast: Cosmopolitanism

Nick, Peter and Fraser discuss what cosmopolitanism is, and how it affects our beliefs about the world. To subscribe to the podcast, add this RSS feed to your preferred player.