How to spin drug prevalence data: a beginners guide

.

Today sees the release of the annual drug use statistics bonanza that is the excitingly titled 'Statistics on drug misuse: England 2007'. For drug policy nerds the report is a treasure trove of information containing a detailed breakdown of prevalence data by age group, region, drug, and a range of other criteria. There is a veritable avalanche of data. This is both good: we need to know this detail because making policy is a complicated business that has to respond in different ways to different populations and patterns of use, and bad: because, if you were, just hypothetically, a politician or civil servant wanting to portray your drug policy in a positive light – you can easily hunt down and cherry pick some data to support your given position. The flip side of course is that if you were an opposition politician, newspaper journalist or crazed blogger wanting to paint a different picture, a less flattering one perhaps, it would be equally easy.

Here’s how it’s done. But lest we forget the backdrop to it all:

Policy maker's bizarre obsession with prevalence measures:
UK signs up to the UN-DPC's 1998 pledge for drug free world by 2008

Reductions in prevalence of drug use are the holy grail of UK drug policy with its rather strange ideological commitment to creating a 'drug free world'. Transform have argued that prevalence is only one of number of indicators for measuring drug policy success, and not an especially useful one at that (problematic use is evidently more useful for starters). But that is another issue, and since the Government have chosen prevalence as one of the key measures for judging the success of drug policy lets have a look. Regards targets the bar has been moved a number of occasions during the lifetime of the 10 year strategy drug strategy. At its launch in 98 the targets were to 'reduce the proportion of people under the age of 25 reporting the use of Class A drugs by 25% by March 2005, and to reduce the proportion of people under the age of 25 reporting the use of Class A drugs by 50% March 2008'. For the 2002 revised strategy these very specific numerical targets had been replaced with the more vague aspirations to 'Reduce the use of Class A drugs and the frequent use of any illicit drug among all young people under the age of 25, especially by the most vulnerable young people'.

Checking in on the Home Office drugs website facts page we see a range of the latest claims for success on various prevalence measures. Aside from the somewhat underwhelming headline:

Class A drug-use among young people is stable, with some significant reductions

There are a number of more specific claims:

• the proportion reporting that they have ever taken any drug has fallen by 16%;
  
  
• the proportion reporting that they have ever taken Class A drugs has fallen by 18%;
  
  
• the proportion reporting the use of any drug in the past year has fallen by 21%;
  
  
• the proportion reporting the use of class A drugs in the past year is stable; and
  
  
• the proportion reporting the use of cannabis in the past year has fallen by 24%

It’s a bit confusing as it’s not immediately clear from the page what 'the proportion' is of, or since when these changes have occurred. Luckily I have in front of me the January Drug Strategy update leaflet containing the same claims, which it turns out relate to changes amongst 16-24 year olds since 1998 as reported in the British Crime Survey. There are two further claims of success, based on data from the Department of Health surveys of 11-15 year olds:

• Use of any drug had decreased: 19% of pupils had taken drugs in 2005, compared to 21% in 2003.
  
  
• Cannabis use had decreased: 12% of pupils had used cannabis in 2005, down from 13% in 2003, 2002 and 2001.

WOW: Impressive. With the exception of class A drug use (the drugs that cause the most harm, you remember; the ones the drug strategy set specific targets to reduce by 50% by next March) which has 'remained stable' (at the highest level in history and the highest amongst any European country), the drug strategy seems to be really delivering.

But let’s delve deep into the bowels of today's report, beyond the rose tinted exec summary and see what we can learn. One of the things you soon spot is that prevalence stats change depending on how you measure them, which populations you look at and also whether you are looking at use in the month, last year, or 'ever used'. You can also present percentages in different ways with quite startlingly different effects.

So from the top of the Home Office prevalence achievements list. From the report we learn that reported life time use of any drug amongst 16-24 year olds has fallen from 53.7 in 1998 to 45.1 in 2005/6. So that 16% fall is actually just over 8%, but it is 16% of the original 53.7 total. Clever.

Similarly, next on down the bullet list, that '18%' fall in reported lifetime use of class A's describes a fall from 20.5 to 16.9 . So that's actually a fall 3.6% (which is 18% of 20.5).

Simlarly, next, the 21% drop in reporting any drug used in past year, turns out to be a drop of 6.4% (31.8 to 25.2). And the 24% drop in last year cannabis use, actually 4.8% (26.2 to 21.4).

Lets be clear: they are not technically lying here, but I feel it's a bit sneaky in that it makes the reported falls in reported drug use (which the Home Office is welcome to celebrate) rather more dramatic sounding than they really are. When i was discussing this with a home affairs journalistrecently he laughed about 'the old percentages of percentages trick'. It rather reminds me of the time last March when the Times, under the ridiculously overdramatic front page headline 'Cocaine Floods the Playground' deployed the same trick to the extreme (but in the opposite direction) when they reported that "Use of the addictive drug by children doubles in a year". It turned out that this doubling – in Home Office statistics: 100% increase in cocaine use - was actually a rise from 1% to 2%. When examined a bit more closely, looking at the non rounded up data revealed the rise was actually from 1.4% to 1.9%. So with a quick wave of the statistical wand the 100% rise becomes a 0.5% rise (which could have been down to sampling error anyway). This particular statistical travesty was covered by Ben Goldacre in the Guardian’s unfailingly excellent Bad Science column here (and now a year has passed and he’s picked up an award for that particular effort I think it’s fair to take my credit for drawing his attention to the story in the first place *takes bow*)

Moving on. If we were to look at some of the other charts in the new publication it would be very easy to show a far less positive picture. Consider for example the equivalent tables from the BCS that look at the broader adult population, from ages 16-59. These show that reported lifetime use of every drug has gone up since 1996 (with the exception of tranquilisers and steroids which fell, and heroin which remained the same - but as we note later the BCS is rubbish at measuring heroin use). This is demonstrated below with the addition (thankyou photoshop) of some colour coded arrows. Look: cocaine use has gone up by 130%!! (well 4.2% anyway).

Personally I think that 'lifetime use' is a particularly un-useful prevalence indicator (although, as we've seen, the Home Office seem happy to chuck it around when it suits them), but actually the 16-59 tables for last year and last month use demonstrate very similar patterns. But you might argue, reasonably, that young people were the primary concern. In that case, to be really alarmist, I might draw your attention to this table based on surveys of drug use of 11-15 year olds (p.43) that I challenge any Home Office wonk to spin into looking even remotely positive:

Nowhere on the Home Office prevalence achievements have I seen the fact that:

The proportion of 11 year olds reporting ever using drugs has risen by 1400% since the start of the drug strategy.

Even though the way they measure this apparently changed in 2001 (with unpspecified effects) its not exactly cause to crack out the champagne is it. Now, returning to the Home Office achievements list again. That stat about pupils reported use of any drug dropping from 21% in 2003 to 19% in 2005. This table would suggest it is true...but not quite the whole story:

If you run from 1998, like they did with all those other stats, the proportion of pupils who have used drugs in the last year has actually increased by 8%, or using the Home Office website percentages technique: 80%. What they have done is compare the 2005 figure to the highest recorded one in the set – blatant cherry picking and potentially very misleading.

The final Home Office stat about cannabis use amongst pupils falling. Well that, I am pleased to say appears to be both correct and properly reported (albeit rounded up/down). In fact in 2006 cannabis use fell again to nearer 10%. Bravo - thats a result (perhaps they should consider reducing penalties on some other drugs?).

So what can we learn from all this.

That drug stats generally are a minefield to be approached with extreme caution, especially when translated through the prism of political spin or lazy journalism. They aren’t that accurate at the best of times – the BCS is perfectly open in conceding it is an underestimate of true drug use because it is a household survey it misses key marginalised populations (where problematic use of heroin for exapmple, is often highest) including those with no fixed address, as well other groups including students living in halls of residence. Add to that the changing statistical methodologies, and the yawning holes in the data collection (illegal activity is generally hard to measure) and the picture, despite the reems of tables, is actually very threadbare.

Still, there is sufficient data that it isn’t too hard to find figures or trends that, with a bit of massaging, show drug use is going up or down, depending on how you want to spin it. Various people, most obviously the custodians of the drug strategy, obviously want to present policy outcomes in best possible light so their ‘best of’ highlights should be taken with a hefty pinch of salt. And to be fair, so should the opposition’s ‘worst of’ lowlights. Look for an independent academic review of the stats like this one, or read them yourself.

Patterns of drug use fluctuate up and down, apparently oblivious to the interventions of Government and enforcement agencies - it is increasingly clear that the key determinants of drug use and misuse (this new document fails to make this important distinction) are socio-economic and cultural ones.

Total Class A drug use appears stable because the fashions for ecstasy and LSD have moved on. But these according to most analyses are two of the least harmful Class A drugs - probably mis-classified by a couple of alphabetic increments. The two drugs the Government has repeatedly said it is most concerned about are heroin and cocaine, because they are responsible for the lion’s share of secondary crime harms – that £16 billion a year in crime costs the Home Office keep mentioning. This is where the real enforcement push has gone, but these are also the two drugs where policy has fared the worst: having risen steadily and dramatically in the case of cocaine, or in the case of heroin, risen up until 2001/2 and then stayed about the same since. There is no good way to spin this – a dramatic rise or stabilisation at a perilous and unprecedented high is NOT a policy success, especially when the key strategc goal of ten years ago – which literally billions of pounds has been thrown at - was a significant reduction. It really doesn’t matter how you dress it up. All the other ‘successes’, the marginal falls in ecstasy, amphetamine, LSD, cannabis use and so on, pale into insignificance next to this monumental disaster where it really matters.