The truth and it's boots: publication bias and shale gas

"A lie can get halfway around the world while the truth is still getting it's boots on". While the origins of this quote are disputed, there can be little doubting of the sentiment behind it.

This can even be true in the peer-reviewed scientific literature: often a "high-impact" finding gets substantial publicity, and is then cited extensively in the literature, while subsequent studies that rebut these findings are, relatively-speaking, ignored.

To be clear, there is no "lying" involved here, in the sense of deliberate misconduct or anything like that. However studies with small sample sizes or especially studies that are poorly designed, are more likely to throw up anomalous results. Once larger studies are performed that are more statistically robust, the anomalous effect, which could have just been a fluke (after all, 95% confidence levels means a 1 in 20 chance of being incorrect), goes away.

This is an important part of science. Smaller preliminary studies may give way to larger studies that produce a more robust result. However, what is important is that the more robust studies are cited as often, or more so, than the one that produced the "sexy" result.

I bring this issue up after reading an interesting blog post here, which considers this issue with respect to educational psychology. An early paper suggested that by making questions on an exam paper harder to read, students would read them more carefully and therefore achieve higher marks.
The study sampled only 40 students. Subsequently, other researchers repeated the study with thousands of candidates, but were not able to repeat the results, finding no difference between test scores regardless of how the question was written.

All well and good, and this is how science should proceed. However the original study, with the result subsequently shown to be incorrect, has been cited hundreds of times and received extensive publicity: it's got halfway around the world - while the subsequent paper, which was much more robust but with a much more prosaic finding - has been cited much less: it's barely got its boots on!


This is analogous to certain papers on shale gas. Papers that claim to find links between shale gas and pollution are far more interesting and scientifically "sexy". Therefore they get widely publicised and cited. Papers that find no links between shale gas and pollution are far more boring, and they fail to get attention. This can be seen in a comparison between several recent papers.

In 2011 a team from Duke University published a paper in PNAS linking shale gas production in the Marcellus to elevated methane levels in groundwater, based on 60 water samples. This paper has been cited over 530 times (Google Scholar). The same team covered the Fayetteville shale in Arkansas in a similar study, but did not find any link between shale gas and groundwater methane. The less-interesting finding was only published in Applied Geochemistry, far less prestigious than PNAS, and has received only 30 subsequent citations.

In 2013 the Duke team published another paper (again in PNAS) again linking methane to drilling in the Marcellus, extending the 2011 study to a total of 140 water samples. Again, the "sexy" result generated substantial interest, and the 2013 paper has been cited almost 150 times. However, also in 2013 a study by Molofsky et al. used almost 2,000 water samples, but did not find any link between groundwater methane and shale gas drilling. Again, this "unsexy" study found it's way into a much lower impact journal ("Groundwater"). With nearly 2,000 water samples vs 140 samples, the Molofsky paper is far more statistically robust than the PNAS papers, yet it has only been cited 50 times.

The impact of this imbalance in publicity has implications for policy-relevant subjects such as shale gas. It is noticeable that recent reports studying the public health impacts of shale gas development, such as the CIEH and Medact reports for example, cite the "sexy" PNAS studies, but fail to cite the more robust Molofsky paper.

To wrap up, publication bias is an acknowledged issue in the academic literature, albeit more so in biological sciences. It is interesting to see it creeping into the geological world. However, I don't really have an easy remedy to conclude with (so suggestions in the comments I suppose).