Tuesday, April 24, 2007

How to spin drug prevalence data: a beginners guide


Today sees the release of the annual drug use statistics bonanza that is the excitingly titled 'Statistics on drug misuse: England 2007'. For drug policy nerds the report is a treasure trove of information containing a detailed breakdown of prevalence data by age group, region, drug, and a range of other criteria. There is a veritable avalanche of data. This is both good: we need to know this detail because making policy is a complicated business that has to respond in different ways to different populations and patterns of use, and bad: because, if you were, just hypothetically, a politician or civil servant wanting to portray your drug policy in a positive light – you can easily hunt down and cherry pick some data to support your given position. The flip side of course is that if you were an opposition politician, newspaper journalist or crazed blogger wanting to paint a different picture, a less flattering one perhaps, it would be equally easy.

Here’s how it’s done. But lest we forget the backdrop to it all:

Policy maker's bizarre obsession with prevalence measures:
UK signs up to the UN-DPC's 1998 pledge for drug free world by 2008

Reductions in prevalence of drug use are the holy grail of UK drug policy with its rather strange ideological commitment to creating a 'drug free world'. Transform have argued that prevalence is only one of number of indicators for measuring drug policy success, and not an especially useful one at that (problematic use is evidently more useful for starters). But that is another issue, and since the Government have chosen prevalence as one of the key measures for judging the success of drug policy lets have a look. Regards targets the bar has been moved a number of occasions during the lifetime of the 10 year strategy drug strategy. At its launch in 98 the targets were to 'reduce the proportion of people under the age of 25 reporting the use of Class A drugs by 25% by March 2005, and to reduce the proportion of people under the age of 25 reporting the use of Class A drugs by 50% March 2008'. For the 2002 revised strategy these very specific numerical targets had been replaced with the more vague aspirations to 'Reduce the use of Class A drugs and the frequent use of any illicit drug among all young people under the age of 25, especially by the most vulnerable young people'.

Checking in on the Home Office drugs website facts page we see a range of the latest claims for success on various prevalence measures. Aside from the somewhat underwhelming headline:

Class A drug-use among young people is stable, with some significant reductions

There are a number of more specific claims:

  • the proportion reporting that they have ever taken any drug has fallen by 16%;

  • the proportion reporting that they have ever taken Class A drugs has fallen by 18%;

  • the proportion reporting the use of any drug in the past year has fallen by 21%;

  • the proportion reporting the use of class A drugs in the past year is stable; and

  • the proportion reporting the use of cannabis in the past year has fallen by 24%

It’s a bit confusing as it’s not immediately clear from the page what 'the proportion' is of, or since when these changes have occurred. Luckily I have in front of me the January Drug Strategy update leaflet containing the same claims, which it turns out relate to changes amongst 16-24 year olds since 1998 as reported in the British Crime Survey. There are two further claims of success, based on data from the Department of Health surveys of 11-15 year olds:

  • Use of any drug had decreased: 19% of pupils had taken drugs in 2005, compared to 21% in 2003.

  • Cannabis use had decreased: 12% of pupils had used cannabis in 2005, down from 13% in 2003, 2002 and 2001.

WOW: Impressive. With the exception of class A drug use (the drugs that cause the most harm, you remember; the ones the drug strategy set specific targets to reduce by 50% by next March) which has 'remained stable' (at the highest level in history and the highest amongst any European country), the drug strategy seems to be really delivering.

But let’s delve deep into the bowels of today's report, beyond the rose tinted exec summary and see what we can learn. One of the things you soon spot is that prevalence stats change depending on how you measure them, which populations you look at and also whether you are looking at use in the month, last year, or 'ever used'. You can also present percentages in different ways with quite startlingly different effects.

So from the top of the Home Office prevalence achievements list. From the report we learn that reported life time use of any drug amongst 16-24 year olds has fallen from 53.7 in 1998 to 45.1 in 2005/6. So that 16% fall is actually just over 8%, but it is 16% of the original 53.7 total. Clever.

Similarly, next on down the bullet list, that '18%' fall in reported lifetime use of class A's describes a fall from 20.5 to 16.9 . So that's actually a fall 3.6% (which is 18% of 20.5).

Simlarly, next, the 21% drop in reporting any drug used in past year, turns out to be a drop of 6.4% (31.8 to 25.2). And the 24% drop in last year cannabis use, actually 4.8% (26.2 to 21.4).

Lets be clear: they are not technically lying here, but I feel it's a bit sneaky in that it makes the reported falls in reported drug use (which the Home Office is welcome to celebrate) rather more dramatic sounding than they really are. When i was discussing this with a home affairs journalistrecently he laughed about 'the old percentages of percentages trick'. It rather reminds me of the time last March when the Times, under the ridiculously overdramatic front page headline 'Cocaine Floods the Playground' deployed the same trick to the extreme (but in the opposite direction) when they reported that "Use of the addictive drug by children doubles in a year". It turned out that this doubling – in Home Office statistics: 100% increase in cocaine use - was actually a rise from 1% to 2%. When examined a bit more closely, looking at the non rounded up data revealed the rise was actually from 1.4% to 1.9%. So with a quick wave of the statistical wand the 100% rise becomes a 0.5% rise (which could have been down to sampling error anyway). This particular statistical travesty was covered by Ben Goldacre in the Guardian’s unfailingly excellent Bad Science column here (and now a year has passed and he’s picked up an award for that particular effort I think it’s fair to take my credit for drawing his attention to the story in the first place *takes bow*)

Moving on. If we were to look at some of the other charts in the new publication it would be very easy to show a far less positive picture. Consider for example the equivalent tables from the BCS that look at the broader adult population, from ages 16-59. These show that reported lifetime use of every drug has gone up since 1996 (with the exception of tranquilisers and steroids which fell, and heroin which remained the same - but as we note later the BCS is rubbish at measuring heroin use). This is demonstrated below with the addition (thankyou photoshop) of some colour coded arrows. Look: cocaine use has gone up by 130%!! (well 4.2% anyway).

Personally I think that 'lifetime use' is a particularly un-useful prevalence indicator (although, as we've seen, the Home Office seem happy to chuck it around when it suits them), but actually the 16-59 tables for last year and last month use demonstrate very similar patterns. But you might argue, reasonably, that young people were the primary concern. In that case, to be really alarmist, I might draw your attention to this table based on surveys of drug use of 11-15 year olds (p.43) that I challenge any Home Office wonk to spin into looking even remotely positive:

Nowhere on the Home Office prevalence achievements have I seen the fact that:

The proportion of 11 year olds reporting ever using drugs has risen by 1400% since the start of the drug strategy.

Even though the way they measure this apparently changed in 2001 (with unpspecified effects) its not exactly cause to crack out the champagne is it. Now, returning to the Home Office achievements list again. That stat about pupils reported use of any drug dropping from 21% in 2003 to 19% in 2005. This table would suggest it is true...but not quite the whole story:

If you run from 1998, like they did with all those other stats, the proportion of pupils who have used drugs in the last year has actually increased by 8%, or using the Home Office website percentages technique: 80%. What they have done is compare the 2005 figure to the highest recorded one in the set – blatant cherry picking and potentially very misleading.

The final Home Office stat about cannabis use amongst pupils falling. Well that, I am pleased to say appears to be both correct and properly reported (albeit rounded up/down). In fact in 2006 cannabis use fell again to nearer 10%. Bravo - thats a result (perhaps they should consider reducing penalties on some other drugs?).

So what can we learn from all this.

That drug stats generally are a minefield to be approached with extreme caution, especially when translated through the prism of political spin or lazy journalism. They aren’t that accurate at the best of times – the BCS is perfectly open in conceding it is an underestimate of true drug use because it is a household survey it misses key marginalised populations (where problematic use of heroin for exapmple, is often highest) including those with no fixed address, as well other groups including students living in halls of residence. Add to that the changing statistical methodologies, and the yawning holes in the data collection (illegal activity is generally hard to measure) and the picture, despite the reems of tables, is actually very threadbare.

Still, there is sufficient data that it isn’t too hard to find figures or trends that, with a bit of massaging, show drug use is going up or down, depending on how you want to spin it. Various people, most obviously the custodians of the drug strategy, obviously want to present policy outcomes in best possible light so their ‘best of’ highlights should be taken with a hefty pinch of salt. And to be fair, so should the opposition’s ‘worst of’ lowlights. Look for an independent academic review of the stats like this one, or read them yourself.

Patterns of drug use fluctuate up and down, apparently oblivious to the interventions of Government and enforcement agencies - it is increasingly clear that the key determinants of drug use and misuse (this new document fails to make this important distinction) are socio-economic and cultural ones.

Total Class A drug use appears stable because the fashions for ecstasy and LSD have moved on. But these according to most analyses are two of the least harmful Class A drugs - probably mis-classified by a couple of alphabetic increments. The two drugs the Government has repeatedly said it is most concerned about are heroin and cocaine, because they are responsible for the lion’s share of secondary crime harms – that £16 billion a year in crime costs the Home Office keep mentioning. This is where the real enforcement push has gone, but these are also the two drugs where policy has fared the worst: having risen steadily and dramatically in the case of cocaine, or in the case of heroin, risen up until 2001/2 and then stayed about the same since. There is no good way to spin this – a dramatic rise or stabilisation at a perilous and unprecedented high is NOT a policy success, especially when the key strategc goal of ten years ago – which literally billions of pounds has been thrown at - was a significant reduction. It really doesn’t matter how you dress it up. All the other ‘successes’, the marginal falls in ecstasy, amphetamine, LSD, cannabis use and so on, pale into insignificance next to this monumental disaster where it really matters.


Bob said...

Less young people using cannabis is certainly a plus point.

But, as you pointed out it is the rise in cocaine which is especially concerning. Many clubbers seem to be swapping from pills to cocaine. Now of course MDMA isnt safe, but it is largely only dangerous to the person using. A cocaine addict is a different animal.

Steve R said...

most club cocaine users arent problematic, or addicts to be fair. I dont think BCS data makes a distinction between cocaine and crack. Crack is obviously far more of a concern as it is much more closely related to chaotic patterns of use, dependence (usually alongside heroin dependence) and offending - its very expensive to maintin a habit. The emergence of the crack 'epidemic' if we can call it that., is one of the disasters that you never see mentioned in the ten year straegy achievements list.

Bob said...

Certainly, crack is quite different in terms of the patterns of use. But, the more casual users there are of cocaine, the more addicts there are likely to be. I'm not of course suggesting that all club cocaine users are sucked in, but a significant minority are.

It does seem very odd that they dont differentiate between crack and cocaine, the Forensic Science Service does when they produce their reports.

Steve R said...

Actually - my mistake - the BCS does make that distinction, although as also discussed the BCS is not a great source of info on problem use of heroin and crack. the MDA doesnt make the distinction and nor do drug-death stats. cocaine and crack are, after all the same drug, just different preparations.

Anonymous said...

Actually, cocaine, when smoked, can be as addictive and leading to impusive behaviour as crack.
The only difference might be price so it's also some kind of discrimination - the poor ones use the cheaper drug so it's a drug of bad guys and the rich ones are good guys by definition of their wealth... is that fair?

Steve R said...

Crack is basically smokable cocaine though isnt it? I thought that was the whole point. My understanding was that if you smoke regular cocaine most of the effect is lost because the heat breaks it down in a way that doesnt happen with crack. Injecting cocaine produces a similar effect to crack i think (ive heard of crack injecting but im not sure what the point is other than perhaps to increase the purity?), but is far riskier - why crack was pure marketing genius from a criminal point of view.

anyway - this is all very off topic - which is, i should remind you, statistical shenanigans.

Bob said...

Yes, crack is the 'base' of cocaine, not the salt. Bases tend to smoke better. But as you say that is very off topic.

Am I right in thinking that the BCS doesnt even talk to the under 16's?

Steve R said...

i dont think they have drug data on under 16s - I'm not sure about the rest but you can find all the data on the home office rds page.

Jill said...

Don't forget that the 11-15 year old data is from school based surveys, which miss out those young people most likely to use and misuse drugs. Interestingly when this survey then looks at alcohol and tobacco use it is only interested in regular and problematic/dependent use, even though alcohol and tobacco use is illegal for this age group.

Danny said...

The thing is with these type of surveys is who exactly are they asking?

Anyone with a brain is not going to admit to illegal activities o a guy in a suit are they?