Science, statistics and society January 21, 2010
Posted by sarah in: science , trackback
What are the odds?
On Tuesday I attended Science Cafe in Leiden, a monthly discussion evening on all matters scientific and their role in society. The theme was the way chance, likelihood and statistics are (mis-)used and represented by the media, politicans and the law. Leading the discussion was Arnout Jaspers, columnist for Dutch science magazine Natuurwetenschap & Techniek, with special guest Richard Gill, Professor in Mathematical Statistics at Leiden University. Gill and Jaspers illustrated the potentially far-reaching consequences of bad statistics with two recent stories to hit the headlines: the reopening of the Lucia de Berk case, and the drug suspension of Germany’s most successful winter Olympian, Claudia Pechstein.
In 2009, Pechstein was banned from competing for 2 years despite never having tested positive for any banned substances. Her blood passport indicated anomalously high red blood cell counts on two occasions, coinciding with major competitions. The values were considered unusual enough to warrant the suspension – the first time that an athlete receives a doping ban based on blood values alone. Somehow, somewhere, a threshold red blood cell count was determined, values above which are considered anomalous enough to be caused only by unnatural tampering with blood. This threshold is purely based on statistics of measured blood values in athletes, rather than medical expertise. Gill and other statisticians have argued that this approach is deeply flawed, and that many athletes will end up being wrongly convicted.
While athletes suspected of doping may not elicit much sympathy in society today, everyone is at risk from bad statistics. Gill talked extensively about one particular case that made waves worldwide, which, the more you think about it, the more shocking it gets. In 2001, nurse Lucia de Berk came under suspicion after an unexplained death on her shift at a children’s hospital in The Hague. Very quickly, the hospital staff became convinced that Lucia had been present for a number of “unexplained deaths” in several wards where she’d worked, and contacted the police. In 2004, Lucia was eventually convicted of seven murders and three further attempts, becoming the Netherlands’ worst ever serial killer.
Key to her conviction was a statistical analysis, performed by a psychologist, predicting the likelihood of one particular nurse being present at such a number of unnatural deaths “by chance” to be 1 in 342 million. Despite all lack of firm evidence, apart from highly circumstantial factors such as cryptic entries in her diary and possible traces of toxicity in some of the victims (later discredited), this number, quoted widely in the media at the time, skewed the opinion of all involved, including that of expert witnesses who changed their medical opinions on the presumption of Lucia’s guilt. Medical professionals asked to give their expert opinion actually ruled deaths previously recorded as natural to be “suspicious”, based on the fact that Lucia was on shift at the time of death. How’s that for a fair trial? This led to Lucia’s conviction and sentencing to life imprisonment in 2006, her sentence repeatedly upheld at appeals.
One very elementary mistake made in the “statistical” analysis is commonly referred to (and it commonly occurs!) the Prosecutor’s Fallacy, in which the likelihood of Lucia, or any defendant, being present at a certain event is set equal to the likelihood of her innocence. So in effect, the “prosecutor” starts their statisctical argument from the presumption that the crime occurred to begin with.
Furthermore, the actual probability calculation was completely mishandled by the “statistician”/psychologist. Gill and colleagues, who took an interest in the case very early on, calculated the likelihood of any given nurse experiencing a sequence of bad luck like Lucia did in 2001 at 1 in 25, or thereabouts – although they argue that the issue is far more complex statistically than this statement would suggest.
After campaigning by Gill and several members of the scientific, mathematical and medical communities in the Netherlands, the Dutch Supreme Court agreed to reopen the case in 2008. In 2009, Lucia was essentially released and she is expected to be formally cleared of all charges in March this year after the prosecution assessed new scientific evidence on the victims’ medical condition. A happy ending perhaps – although how a person recovers from being branded their country’s most brutal serial killer and several years’ imprisonment despite being completely innocent – fuck knows.
If you want to mull over the details of the case – and do, it’s fasinating reading – I’ve added some links below.
In astronomy, bad statistics can make for a crappy paper, or a bad result that will be refuted or contradicted by subsequent analyses. But that’s not where it ends. In society today, increasing amounts of data are gathered every single day on all of us, and these data are routinely searched and cross-checked for outliers or specific patterns of activity. In effect, in the same way that athletes have biomedical passports to check up on the purity of their blood, we all have “behavioural passports”, and too often suspicion carries an automatic presumption of guilt.
Who decides what the thresholds are for anomaly, and on what basis? Can we trust that these standards are set in a statistically responsible way?
What amazed me a little at the Science Cafe discussion is the audience’s fascination with statistical statements of little consequence. What does “a 30% chance of rain” actually mean? Should I take an umbrella? What’s the likelihood of my bike getting stolen an 8th time? These are interesting questions, but compared with the danger of being convicted of a crime on the basis of some dodgy numbers, pretty trivial. It seems the well-heeled burgers of Leiden feel very far removed from situations such as de Berk’s and Pechstein’s.
Yes, Lucia’s name has been cleared – but the fact that this horrific case occurred in the first place in a liberal country like the Netherlands, so big on freedom of speech and civil liberties, is firm evidence that none of us are truly safe from bad statistics.
Reading
Richard Gill’s home page with links to papers and opinions
Official webpage for Lucia de Berk’s case
Opinion and analysis by Piet Groeneboom, statistician
Ben Goldacre’s post from 2007 describing the errors in the statistical calulcation, with good discussion in the comments. [Update: Goldacre also discusses this case in his book
Stories on Lucia in the International edition of the NRC Handelsblad
Comments»
[...] This post was mentioned on Twitter by Sarah Kendrew, Jacquelynn Potter. Jacquelynn Potter said: Never held much truck with statistics anyway! RT @sarahkendrew New blog post, Science, statistics and society – http://tinyurl.com/y9ng4tc [...]
[...] book also reminded me of the tragic case I wrote about earlier this year, in which a Dutch nurse was convicted of multiple murders on the basis of botched [...]
[...] update from the frontlines of judicial excellence. As expected, nurse Lucia de Berk was cleared of all murder charges by the court of Arnhem on 14 April last week. The case has been [...]