Sigmas and Sharpshooters

Today's paper is a recent report published in the journal "Earth's Future" looking at methane emissions from shale gas operations in the USA. As you'd expect from a paper that is critical about shale gas exploration, it has received extensive media coverage.

However, the paper falls short in a couple of really important ways, which I'll discuss below. Sadly, it provides a few handy lessons about how not to go about doing science. The first issue is falling foul of the Texas Sharpshooter Fallacy, the second is failing to use the proper measures to ensure the result is statistically significant.

Firstly, the Texas Sharpshooter Fallacy. The parable is of a hopeless Texan gunman looking to prove to the world his martial prowess. So he takes aim with his pistol at the side of a barn, and blasts away. Once he has done shooting, he notices that by chance some of his shots happen to have hit close together. He then paints on a target with its bullseye at that point, before inviting the neighbours over to admire the results of his sharpshooting skills.

More technically, this fallacy describes a situation where certain clusters of data are cherry picked from a larger population because they happen to fit your hypothesis, ignoring all the cases that would disprove the hypothesis.

So how does this fallacy apply to the paper in question? The image below shows the methane measurements for 2006-2008 (the "before" case) and 2009-2011 (the "after" case) presented in the paper:

It's clear that methane has gone up substantially all across the USA in this period. There are many sources of methane emissions, both naturally occurring (bogs, swamps etc) and man made (farms, coal mines, conventional gas wells, and shale wells). What is noticeable is that while there are places where there is shale gas activity and high methane concentrations, there are plenty of places with no oil and gas activity that have seen methane levels rise, while in other places there is shale gas activity but methane levels that are not particularly relevant.

For example, Nebraska saw substantial increases in methane, yet in 2010 there were only 2 drilling rigs in the entire state. It's a similar story in, for example, Iowa (0 drilling rigs), Illinois (2 drilling rigs) and Indiana (3 drilling rigs). In contrast, Arkansas, home of the Fayetteville shale with 39 active rigs in 2010, and Northwestern Louisiana, home of the Haynesville shale with 135 active rigs in 2010, have noticeably low methane concentrations.

There are many different shale gas/oil plays across the USA. It is apparent that methane concentrations also vary across the USA. It is therefore inevitable that, just by chance, some areas of high methane will correlate with areas of shale production. Our sharpshooters have drawn their targets around 3 such areas (the black boxes in the above image) and declared themselves to be expert marksmen. Not good science.

We can see the same effect within the individual study areas as well. The following image shows the change in methane levels for Texas from 2006-2008 to 2009-2011:

During this time, there was active drilling and unconventional hydrocarbon production from the West Texas Permian Basin, the Haynesville Shale and the Eagle Ford Shale. Neither the Permian nor the Haynesville show anything out of the ordinary, while there are other areas with no active drilling that have seen substantial methane increases. It's a similar story for the Marcellus in Pennslyvania, shown below: there are places with drilling that have high methane levels, but also places with drilling that have low measurements, and places with high measurements that do not have drilling.

The second issue is one of error bars and confidence intervals. With any scientific measurement, there is an error bar marking the interval over which we can be confident the result is accurate. Typically, confidence limits of 95% are used - if it is said that a measurement is 5 ± 1.5 at a 95% level, then we can be 95% confident that the true value lies somewhere between 3.5 and 6.5.

The authors of this paper complete their analysis for the Bakken and Eagle Ford shales, concluding that methane emissions have increased by 990 ± 650 ktCH4/yr and 530 ± 330 ktCH4/yr in each case.

What is unusual, however, is the limits they have chosen for their error bars. These are set to the 1-σ level, or one standard deviation. This corresponds to a confidence interval of only 68%, meaning there is a 1-in-3 chance that the computed value was arrived at by chance.

Scientists generally use the 2-sigma level as an error bound - corresponding to a 95% confidence level in the result (which still means that the measured observations could have occurred by pure chance 1 time in 20). For really important experiments, scientists will require even higher confidence bounds, like the 5-sigma bound for the Higgs Boson discovery, which means a 1 in 3,500,000 chance of a spurious result.

I've not often seen a confidence level of 1-sigma being used in peer reviewed science, given the implication of a 1-in-3 chance of being a spurious result. Instead, let us double their confidence levels to the 2-sigma limit (95%) more normally expected as a minimum for scientific findings. We then find the results have become 990 ± 1300 ktCH4/yr and 530 ± 660 ktCH4/yr. In both cases the error bars have become larger than the values themselves. We cannot even be sure whether rates of methane emissions have increased or decreased, since the lower error bars at the 95% level fall below zero.

In short, even with the Texas-Sharpshooting described above, the authors have not managed to produce statistically robust evidence to back up their claims. However, it's given me a chance to discuss both the Texas Sharpshooter Fallacy (which is also a common problem in attempts to forecast earthquakes) and the importance of error bars, which I am sure both scientific and non-scientific readers alike will have enjoyed.

Europe's Geological Surveys Make Joint Statement on Shale Gas

After a meeting in Copenhagen, the North Atlantic Group of the European Geological Surveys released a statement on shale gas and fracking. More info is available here.

This group is made up of the Geological Surveys of the UK, Germany, Ireland, Holland, Norway, Iceland and Denmark, so it represents most of Europe's geological expertise. Make no mistake, these guys are experts.

Below are a choice selection of quotes both from their Copenhagen statement, as well as comments from the President of Germany's Federal Institute for Geosciences (the BGR) in a press release. The press release is in German, so I have used google to translate as best I can. Apologies to any German speakers who spot any mis-translations, but the general gist of the statements should be clear to all.

Firstly from the Copenhagen Statement:

"The Survey Directors are concerned that frequent misleading media messages regarding exploration and exploitation of raw materials and geo-energy have the potential to obscure scientific results and conclusions, and may ultimately lead to poor decisions for Society."

"The Survey Directors emphasise that their Surveys hold the majority of key sub-surface data for their Nations. They are thus best placed to objectively and independently inform decision makers, on shale and other georesource estimation exercises, and on some of the potential environmental risks of the operations."

"Particular concern was expressed that the role of the national Geological Survey may be bypassed, resulting in the submission of poorly formulated geoscientific advice to governments."

The press release from the BGR was even more explicit:

"The European Geological Surveys of the North Atlantic area, which include the Federal Institute for Geosciences and Natural Resources (BGR), fear in the face of misleading reports in the media about the exploration and extraction of raw materials negative socio-political consequences."

"Often dangers are evoked where there are none. When fracking for production of natural gas there are widespread fears in the population, most of which are unfounded from geoscientific perspective."

"Since the early 1960s, more than 320 fracking measures in conventional natural gas production have been carried out in Lower Saxony. The technique used here is similar to the method for the development of shale gas resources. In these past operations by fracking there has not been a single incident in which the environment has been damaged. When critics speak in connection with fracking as an uncontrollable high-risk technology, this is just wrong under scientific criteria." (my emphasis)

I think these comments will be worth remembering as the shale gas debate rumbles on over the coming months.

Does the infrastructure bill give carte blanche to inject any substance an operator chooses?

Today's fuss is over the infrastructure bill currently going through parliament. The purpose of this bill is to reduce the amount of time spent in court arguing over subsurface access rights and trespass issues.

However, anti-fracking activists have today tried to claim that the bill is an attempt to subvert existing regulation, allowing operators to inject whatever fluid they want without any safeguards. The Guardian has some typically scaremongering coverage here.

Greenpeace are claiming that

"Ministers are effectively trying to absolve fracking firms from responsibility for whatever mess they’ll end up leaving underground"

while Friends of the Earth claim that

"The government appears to be trying to sneak through an amendment which would allow fracking firms to reinject their waste under people’s homes and businesses" 

Are they right? Of course not.

Advertising Complaints in Australia

A few months ago I blogged about Frack Free Somerset (FFS)'s decision not to challenge a complaint about their promotional material made to the Advertising Standards Agency (ASA). Their leaflet contained numerous errors and misleading statements. Because FFS agreed to remove their material and cease using it, rather than to attempt to offer a rebuttal, the ASA never carried out an investigation.

There are obvious parallels to the complaint made to the ASA over Cuadrilla's publicity material, and over an advert placed by the self-styled "Frack-Master" Chris Faulkner.

We in the UK are not the only country to provide an advertising regulator, and in this post I will report on a decision reached by the Publishing Advertisers Bureau in Australia.

Before I do so, however, I want to comment on the situation we now find ourselves in, where advertising standards agencies are finding themselves having to make judgements on what are, in some cases, quite complicated and technical issues, with very little understanding of the subject matter. I very much doubt that anyone at the ASA has any familiarity with oil and gas operations and/or regulation.

While I am sure that the ASA are used to dealing with complaints in subject areas they are not familiar with, I would suggest that this debate is very different to determining whether or not a new brand of shampoo really makes your hair feel 10 times silkier. Yet as an authoritative body it is inevitable that their pronouncements are taken very seriously indeed, when the more I think about it, the less reason I see to do so. Frack-Master Chris Faulkner summed the situation up: "the ASA has been both judge and jury in this case. They appear to have become unqualified experts in fracking and interpreting the complex issues surrounding fracking in the UK".

However, today's blog is about a decision in Australia. The opposition group Frack Free Geraldton (FFG), with support of the Conservation Council of Western Australia (CCWA), published an advert in the local rag, the Geraldton Guardian. The Australian Petroleum Production and Exploration Association (APPEA) submitted a compliant regarding the advert, which appears to have been upheld. In each case, the statements made by FFG and CCWA were found to be misleading and deceptive.

The statements, and the reasons for the findings, are discussed below.

"Shale fracking, the process of extracting gas by using toxic chemicals to crack deep rocks, can turn our water into a dangerous chemical cocktail"

It was found that this statement gives a misleading impression of the fracturing process, because it gives the impression that most of the fracking fluid is composed of toxic chemicals. It was found that "The statement that 'toxic chemicals' are used to crack deep rocks creates the impression that toxic chemicals 'alone', certainly not in such small percentage quantities are used to frack", which is not the case: frack fluid is 99% water, with only a small amount of additive, most or all of which is not toxic.

The statement finds that "to an ordinary reasonable reader the words of the advertisement and the accompanying illustration together create the impression that the amount of 'toxic chemical' used is a much greater concentration that is in fact the case", which is a misleading and deceptive exaggeration.

With regards to turning water into "a dangerous chemical cocktail", it was found that while there are risks posed by hydraulic fracturing, "the consensus of scientific data suggests that there have been no cases internationally of hydraulic shale gas fracturing inadvertently breaching a water source and thereby causing contamination", and that "a combination of research from around the world shows us that the risks are low".

Moreover, in their response to the complaint, CCWA "have not produced any evidence that hydraulic fracking fluid has in the course of any hydraulic shale gas fracking process permeated a fresh water aquifer. Its contentions are against the scientific literature".

"Research in the US has found that 6% of fracking wells leak into ground water in the first year"

Anyone who is familiar with this blog will already know why this statement is misleading. It is a topic I have discussed extensively. The 6% statistics refer to the number of wells that have some kind of casing or cement issue in one of the casing strings. However, wells have several casing strings to separate the production zone from any sensitive groundwater supplies. This means that a well with an issue in one casing string will not be spewing hydrocarbons into the environment. It's a belt-and-braces type approach.

A paper by King and King in 2013 (SPE) is instructive in this regard:

For US wells, while individual barrier failures (containment maintained and no pollution indicated) in a specific well group may range from very low to several percent (depending on geographical area, operator, era, well type and maintenance quality), actual well integrity failures are very rare. Well integrity failure is where all barriers fail and a leak is possible. True well integrity failure rates are two to three orders of magnitude lower than single barrier failure rates.

In their response, the CCWA admit that their statement "is not materially correct", and it is therefore found to be "misleading and deceptive".

This is a point I've been making for some time, so it's good to see that even environmental bodies know that it is not correct to say that 6% (or 30% or 50% or whatever) of wells are leaking, even if they do still insist on claiming this in their promotional material.

"Once our water is contaminated, it will be forever" 

This statement ignores the abundant evidence that while any contamination incident is bad, the damage is rarely permanent: wells that do leak can be repaired, spills can be remediated. For example, Considine et al. (2013) examine the environmental impacts from drilling in Pennsylvania, and find 25 incidents that they deem to be "serious". However, they find that in all but 6 cases the impacts had already been remediated satisfactorily. The APPEA provided similar examples of remediation in their supporting evidence for their complaint.

In contrast, "the contentions put forward by CCWA in support of this statement in its submission dated 22 August 2014 are without any independent scientific support. They are unsupported assertions." 

As a result, it was found that "the statement that once contaminated water will forever be contaminated is not supported by contemporary scientific views and is misleading and deceptive." 


For anyone interested in the original complaint, the CCWA rebuttal and the final decision, the documents are available below.

The original APPEA complaint is here.
The CCWA response is here.
Further APPEA comments are here.
The final decision is here.

The CIEH and me: full discussion

A few months ago, the Chartered Institute of Environmental Health (CIEH) released a report into the impacts of hydraulic fracturing and shale gas extraction in the UK. You can read the original report here. Of particular concern was that the report's lead author had already taken a leading role in protests outside of unconventional gas drilling sites in the UK: hardly the best place to start if you are looking for a balanced report.

In my opinion, the quality of the report was very very poor, ranking as little more than scaremongering and with little understanding of the oil/gas extraction process. I was infuriated enough to write a rather outspoken blog post criticising the report, which some readers might remember.

In the days following the report's publication, I ordered my thoughts a little from my original post, and sent a more moderate and thoughtful criticism to the CIEH. The CIEH provided an initial response to my criticisms.

Incidentally, the CIEH made their response public, without actually stating my criticisms alongside their response (or informing me that they had done so). I can only assume that they weren't comfortable seeing their response alongside my original criticisms.

This wouldn't surprise me given the weakness of the response. In some places their is direct and/or tacit admission of error, while in others the authors manage to contradict themselves. I was originally prepared to let sleeping dogs lie and "agree to disagree", especially since I had a busy fieldwork schedule over the summer. However, the quality of the response was so poor that I couldn't help but write a further comment to the CIEH. I have not now received a response to those comments (nor do I expect to, if I'm honest), so I have decided to make public the full extent of our discussion for all to see.

I should add a warning, these are fairly lengthly documents, but hopefully make for an interesting read if you have a few minutes to spare and a cup of tea to hand.

My first critique of the CIEH report is here.

The CIEH response to my critique is here.

My comments on the CIEH response is here.