Sigmas and Sharpshooters

Today's paper is a recent report published in the journal "Earth's Future" looking at methane emissions from shale gas operations in the USA. As you'd expect from a paper that is critical about shale gas exploration, it has received extensive media coverage.

However, the paper falls short in a couple of really important ways, which I'll discuss below. Sadly, it provides a few handy lessons about how not to go about doing science. The first issue is falling foul of the Texas Sharpshooter Fallacy, the second is failing to use the proper measures to ensure the result is statistically significant.

Firstly, the Texas Sharpshooter Fallacy. The parable is of a hopeless Texan gunman looking to prove to the world his martial prowess. So he takes aim with his pistol at the side of a barn, and blasts away. Once he has done shooting, he notices that by chance some of his shots happen to have hit close together. He then paints on a target with its bullseye at that point, before inviting the neighbours over to admire the results of his sharpshooting skills.

More technically, this fallacy describes a situation where certain clusters of data are cherry picked from a larger population because they happen to fit your hypothesis, ignoring all the cases that would disprove the hypothesis.

So how does this fallacy apply to the paper in question? The image below shows the methane measurements for 2006-2008 (the "before" case) and 2009-2011 (the "after" case) presented in the paper:

It's clear that methane has gone up substantially all across the USA in this period. There are many sources of methane emissions, both naturally occurring (bogs, swamps etc) and man made (farms, coal mines, conventional gas wells, and shale wells). What is noticeable is that while there are places where there is shale gas activity and high methane concentrations, there are plenty of places with no oil and gas activity that have seen methane levels rise, while in other places there is shale gas activity but methane levels that are not particularly relevant.

For example, Nebraska saw substantial increases in methane, yet in 2010 there were only 2 drilling rigs in the entire state. It's a similar story in, for example, Iowa (0 drilling rigs), Illinois (2 drilling rigs) and Indiana (3 drilling rigs). In contrast, Arkansas, home of the Fayetteville shale with 39 active rigs in 2010, and Northwestern Louisiana, home of the Haynesville shale with 135 active rigs in 2010, have noticeably low methane concentrations.

There are many different shale gas/oil plays across the USA. It is apparent that methane concentrations also vary across the USA. It is therefore inevitable that, just by chance, some areas of high methane will correlate with areas of shale production. Our sharpshooters have drawn their targets around 3 such areas (the black boxes in the above image) and declared themselves to be expert marksmen. Not good science.

We can see the same effect within the individual study areas as well. The following image shows the change in methane levels for Texas from 2006-2008 to 2009-2011:

During this time, there was active drilling and unconventional hydrocarbon production from the West Texas Permian Basin, the Haynesville Shale and the Eagle Ford Shale. Neither the Permian nor the Haynesville show anything out of the ordinary, while there are other areas with no active drilling that have seen substantial methane increases. It's a similar story for the Marcellus in Pennslyvania, shown below: there are places with drilling that have high methane levels, but also places with drilling that have low measurements, and places with high measurements that do not have drilling.

The second issue is one of error bars and confidence intervals. With any scientific measurement, there is an error bar marking the interval over which we can be confident the result is accurate. Typically, confidence limits of 95% are used - if it is said that a measurement is 5 ± 1.5 at a 95% level, then we can be 95% confident that the true value lies somewhere between 3.5 and 6.5.

The authors of this paper complete their analysis for the Bakken and Eagle Ford shales, concluding that methane emissions have increased by 990 ± 650 ktCH4/yr and 530 ± 330 ktCH4/yr in each case.

What is unusual, however, is the limits they have chosen for their error bars. These are set to the 1-σ level, or one standard deviation. This corresponds to a confidence interval of only 68%, meaning there is a 1-in-3 chance that the computed value was arrived at by chance.

Scientists generally use the 2-sigma level as an error bound - corresponding to a 95% confidence level in the result (which still means that the measured observations could have occurred by pure chance 1 time in 20). For really important experiments, scientists will require even higher confidence bounds, like the 5-sigma bound for the Higgs Boson discovery, which means a 1 in 3,500,000 chance of a spurious result.

I've not often seen a confidence level of 1-sigma being used in peer reviewed science, given the implication of a 1-in-3 chance of being a spurious result. Instead, let us double their confidence levels to the 2-sigma limit (95%) more normally expected as a minimum for scientific findings. We then find the results have become 990 ± 1300 ktCH4/yr and 530 ± 660 ktCH4/yr. In both cases the error bars have become larger than the values themselves. We cannot even be sure whether rates of methane emissions have increased or decreased, since the lower error bars at the 95% level fall below zero.

In short, even with the Texas-Sharpshooting described above, the authors have not managed to produce statistically robust evidence to back up their claims. However, it's given me a chance to discuss both the Texas Sharpshooter Fallacy (which is also a common problem in attempts to forecast earthquakes) and the importance of error bars, which I am sure both scientific and non-scientific readers alike will have enjoyed.