Numbers and statistics can easily confuse - sometimes deliberately, often by accident. Averages can be chosen to prove a point, targets selected so they can't be missed, random clusters of events given a significance they don't deserve, and corrections for confounding either ignored or used to draw conclusions where none are justified. Nigel Hawkes provides examples of all these statistical tricks and explains how to avoid being taken in by them
Speakers: Nigel Hawkes (Straight Statistics)
Nigel Hawkes is a science and health journalist. He graduated from Oxford with a degree in metallurgy in 1966, and has written about science, health and international affairs in a career that began on the staff of Nature and included long spells at The Observer (1972-90) and The Times (1990-2008). He retired from The Times in 2008 after eight years as Health Editor, and is now a columnist for British Medical Journal and Director of a new pressure group, Straight Statistics, which campaigns for the honest presentation and use of statistical data by government, media, and others.
He has written a number of books, including Structures, a book about building and civil engineering, and more than 40 science and technology titles for children and teenagers. He was appointed CBE in 1998 for services to the newspaper industry and science, and was the Medical Journalists Association health writer of the year in 2007.
Steven Forrest, Royal Holloway University
The range of statistics collected by the UK government has changed markedly from details of material wealth in the Doomsday Book of 1086 to the 30 page census in 2001, but do we trust those that make use of statistics?
Nigel Hawkes showed statistics from a 2009 survey which revealed that 60% of participants disagreed with the statement the "government presents official figures honestly" with only 1% strongly agreeing with that statement. The survey showed similar levels of public mistrust in newspapers use of statistics. Hawkes then went on to say that these levels of mistrust were higher than those of other European countries and suggested that this could be due to government lies in the past.
So how do journalists engender public trust in the statistics they use? Well start by making sure that sources are reputable and explain the situations that they apply too.
Nigel Hawkes recommended sourcing statistics from the UK National Statistics website although he said that instead of trawling their website use google, adding "ONS" in the search bar. Hawkes uses this to check "common public knowledge" such as that of binge drinking increasing in recent years. After looking at the statistics the common assumption about drinking is shown to be a "complete illusion". There has actually been a decrease in the units of alcohol consumed. Where does my money go? Is another website recommended as is Open Data – both of which show public spending by the government, providing accurate figures for journalists.
There are many ways that statistics are used in order to create a story. Nigel Hawkes highlighted several techniques to look out for:
Using big numbers. This is done to create a sense of wonder in the audience but is the number credible, how is it calculated and is it inflated and or an extreme value. Big numbers make headlines: "308,889 children each year injured trying to stop arguments between their parents". This seems like shocking news, but as Hawkes dug a bit deeper he found that the charity only asked 1000 children in this survey. Approximately 75 answered that they had been injured in this way and this figure was then extrapolated based on the number of children in this age group in the country to get such a sensational figure.
Misleading statistics. Hawkes looked at the size in the gap between the earnings of men and women, a topic which actively engaged the audience during his session. The Office for National Statistics (ONS) states that women are paid 12.8% less, whilst the Trade Union Congress (TUC) found it to be 37% less. To the surprise of the audience Hawkes then proceeded to say that both of these figures were right and that it was what they were actually comparing which differed greatly. The ONS compared full-time and part-time employees, whilst the TUC compared full-time male employees to part-time female employees. (The ONS also found in a comparison between part-time employees that women were paid 3.4% more than men.)
In contrast some statistics are formed from statistically worthless data. Newspaper polls use self-selected samples – normally a section of the public who read the same newspaper and have similar views. Some agencies exist to compile statistics from three to four thousand 'randomly-chosen' participants but these are statistically worthless as they are not representative of the public.
Mismatched Framings. This is widely seen in medical journals with the benefits of a particular treatment strategy for example, being presented as a relative risk and the drawbacks through absolute measures. This allows statistics to be framed so that the benefit looks greater.
Confounding Factors also play a part and are other factors affecting the statistics. These can be corrected but in some cases it can completely reverse the balance of the argument.
Overall we should not discard statistics from media coverage, but they need to be used responsibly and journalists should be wary of sensationalised statistics used to attract readers. As Nigel Hawkes concluded "They (statistics) are a tremendous tool for journalists. But they can also be a tool of misrepresentation and spin which we ought to be sophisticated enough to see through – and with little effort, we can".