How can journalists be constructive critiques of peer review? What role do bloggers have in creating their own form of peer review?

Speakers: Peter Aldhous (New Scientist), Philip Campbell (Nature), Brian Deer (freelance investigative reporter), Fiona Godlee (Editor, BMJ), Tracey Brown (Director, Sense About Science). Chair: Connie St.Louis (City University).

Session Review

Roosa Sofia Tikkanen, University of Manchester

Peer review is considered the 'ultimate stamp of validity' of scientific publishing, according to Peter Aldhous of New Scientist. Indeed, Philip Campbell, Editor-In-Chief of Nature, would go so far as to say that the entire reputation of the journal 'depends on peer review'. Thus it is safe to say that peer review is the cornerstone of science. But how is the system changing, and is change needed?

Philip Campbell was the first to admit that subjectivity necessarily plays a part in the peer review process. To this end, Fiona Godlee, Editor of the British Medical Journal (BMJ), pointed out that the current system is 'flawed in a million ways'. In fact, Godlee openly admitted that in the BMJ, 'treatments are often presented as safer (and) more reliable than they actually are'. As examples Godlee mentioned Vioxx, a widely prescribed pain and arthritis drug marketed by Merck which was withdrawn due to safety concerns, as well as the recent case of oseltamivir (Tamiflu), which Godlee admitted, even to this day, 'we do not know whether it works'. Not surprisingly, Godlee affirmed that commercial forces from the pharmaceutical industry are always at work in such cases.

Peter Aldhous continued this line of thought by introducing a case of falsification in a 2007 Nature paper. Stem cell scientists at The University of Minnesota, Minneapolis duplicated their data not only within the original paper, but also in another manuscript published in Experimental Haematology. Aldhous stated that such cases of 'faulty methodology' combined with 'overenthusiastic interpretation' by reviewers contribute to peer review being 'biased, unjust and unaccountable'.

In contrast, Tracey Brown, Director of Sense About Science, praised the current peer review system in having 'all the benefits of organised human judgement'. Brown was convinced that the 'myth busting approach' of peer review is the only way that we 'stand a chance of discovering and benefiting from scientific discoveries'.

In a talk entitled 'Monkey Brain Magic: Peer Review in Cyberspace', Brian Deer of the Sunday Times highlighted some of the benefits that science journalists can reap from online post-publication peer review. This newly emerged, more informal type of peer review takes place online through a wiki-style judging of a research paper by the scientific community after its release. Deer stated that internet blogs 'can tell you where the dead bodies are buried'. To accentuate this point, Deer used a fresh example of a paper published in Acta Neurobiologicae Experimentalis that made a case for the measles, mumps and rubella (MMR) vaccine causing changes in brain development in infant macaque monkeys, similar to the abnormal brain growth observed in autism. Within a couple of days of publication, bloggers had picked up on several inconsistencies, including data omission from analysis and serious conflicts of interest amongst authors. Deer emphasised that specialist science bloggers are often better at spotting such errors than journalists, thus providing a shortcut to fact-checking. 'Peer review has become a more dynamic process thanks to the internet', Deer insists. Fiona Fox, Director of the Science Media Centre, called this 'a new breed of citizen critics'. Philip Campbell recognised that whilst the internet is 'making peer review a more transparent and universal process', post-publication review could never replace the current system. 'There is no way in which this will work', Campbell declared.

So how can we begin to address the shortcomings of peer review to ensure falsified data does not get published? Aldhous stressed that the key to successful peer review is to critically look at the available data. However for journalists this task is made more 'difficult' when there is a lack of access to raw data. In fact, Godlee encouraged journalists to campaign for access to raw data in order to disseminate 'the truth'.

But hold on. If we cannot trust the data presented in journals to be accurately analysed and represented, does this mean that there is no trust left in the current scientific publishing system? Clearly peer review is changing, not least because of the internet. It was reassuring to see cases of misconduct brought up during the session, as with increasing awareness and protesting against such misbehaviour, we are certainly on our way to improving the state of peer review.