Wednesday, March 09, 2011

How to skew polls and influence people

Most people would assume that professional pollsters understand the importance of how their questions are worded. If you want to obtain valid results from your poll, you need to make sure that your questions represent both sides of an issue fairly, without any latent bias in their wording.

For example, if you want to know people’s views on taxation, you should recognise that there is likely to be a vast difference in responses between the following questions:

(1) Do you favour an increase in the level of tax you pay on your income?
(2) Do you favour greater investment in public services such as healthcare, education and policing?

Obviously the two questions are different sides of the same coin, but if you only ask the first, you’re likely to get an overwhelming ‘no’, and if you only ask the second, you’re likely to get an overwhelming ‘yes’. Predictably, therefore, how a question is phrased, or the context in which it is asked, affects the answer you will get to it (as ‘Yes, Prime Minister’ brilliantly satirises in the scene below).

Pollsters appear to understand this concept when it comes to most issues – but rarely with drug policy reform.

If you look at the findings of an Angus Reid public opinion poll conducted last year, then you may feel disheartened at the lack of public support for a more progressive UK drug policy. In that poll, participants were shown a list of banned substances, including cannabis, along with the question ‘Do you support or oppose the legalisation of each of the following drugs?’.

Unsurprisingly, when formulated bluntly like this, the question yielded only a 35% level of support for the 'legalisation' of cannabis; and support for the legalisation of other drugs (generally seen as riskier) was lower still, with approximately one in five respondents approving of such a move. (It should be noted, however, than even this seemingly low level of support is relatively encouraging, as several years ago some polls claimed that only one in ten were in favour of legalising the most risky drugs.)

Part of the problem relates to the visceral aversion, embedded in the public consciousness, to the word ‘legalisation’. This aversion is largely based on confusion about what the term means when used in the drugs debate. The confusion is not surprising; it stems from the fact that legalisation is merely a process (broadly of making something illegal, legal), rather than a policy end point. A straight ‘Legalisation: yes/no?’ question gives no indication of how the legal regulatory regime being advocated as the final outcome of the process might actually work.

In the absence of such policy context just saying ‘legalisation’ on its own can reasonably be taken to suggest the removal of all controls – moving to the sort of commercial free market that Transform and most drug law reform advocates are specifically not calling for.

The legalisation question without any policy or regulatory context can also be confused with a question about personal or moral approval of drugs or drug use (in effect, ‘Do you approve of/condone the use of “drug x”’), as opposed to the real question, which is about what one thinks is the best policy response to dealing with a particular drug or drugs in society. We may, for example, disapprove of unhealthy food or overeating without suggesting blanket prohibitions on pork scratchings, or criminal punishments for people who eat too many of them.

If we want to know whether people morally approve of certain drug-using behaviours then it would be fine to ask a question about that. If we want to know what form of legal regulation people think would be appropriate for certain drugs or drug-using behaviours then we need a better question than ‘Legalisation: yes/no?’.

As a starting point, instead of using only the word ‘legalisation’ in opinion polls on drug policy reform, it might be more appropriate and accurate to ask people whether they support or oppose ‘legal regulation of drug production supply and use’, or ‘legalisation and regulation’, which would be better, if still imperfect. What is really needed, however, is a more specific and detailed description of the policy options people are being asked to chose from.

This semantic minefield of drug policy terminology is made all the more perilous by misunderstandings of the word ‘decriminalisation’. Instead of conflating the meanings of ‘decriminalisation’ and ‘legalisation’, as so often happens in media debate, pollsters should be aware of and clearly clarify the distinction between the two terms. The term ‘decriminalisation’ is usually understood to refer to the removal of only criminal penalties for certain activities (most commonly possession and personal use of drugs), but not of other, non-criminal sanctions, such as fines. 'Legalisation', by contrast, refers to a transition from prohibited to legally regulated production, supply and availability, with decriminalisation of use implicit in this process.

Neither term, unfortunately, has a strict legal definition, so they are subject to frequent confusion, often being used interchangeably. The only solution to this misunderstanding is to refrain from using the words in isolation and, again, contextualise them with some clear and concise explanatory text, eg: ‘Decriminalisation – moving from criminal to civil/administrative sanctions such as fines, for personal possession and use’.

When you look at polls which are more aware of the nuances of drug policy language, the results are strikingly different and give reason to be considerably more optimistic about the prospects for reform.

A recent poll of 2,000 people, commissioned by the Liberal Democrats for Drug Policy Reform, asked participants to select the most tolerable of three regulatory options for a range of drugs, along with a clearer description of what each option actually entails. They were:

  • Light regulation (drugs sold like tobacco and alcohol are now)
  • Strict government control and regulation (an example of how government could heavily regulate a legal market in an attempt to minimise harm)
  • Prohibition (the current status of illegal drugs)
Now without claiming this formulation is methodologically bulletproof, it does demonstrate that when given a clearer overview of the features of each option, it seems that, contrary to the findings of the less comprehensive Angus Reid poll, respondents are increasingly receptive to the idea of moving from absolute prohibitions to some method of legally regulating drug markets and availability.

The results of the LDDPR poll were that 70% of participants favoured some form of legal regulation of cannabis, with one in three of those polled favouring a regulatory system similar to that for tobacco and alcohol.

It also emerged that, when compared to the results of the Angus Reid poll, a far greater number of people would like to see the legally regulated availability of ecstasy (39% vs. 19%), cocaine (36% vs. 16%) and heroin (30% vs. 18%).

Perhaps emphasising just how much of a difference good polling can make, the LDDPR survey also made the somewhat unexpected finding that Daily Mail and Daily Express readers constituted the demographic most in favour of the strict control and regulation of drugs. Total support for at least some system of legal regulation was 66% among these readers, and 67% among Conservative voters.

Clearly more work in this area is needed (perhaps with some independently agreed formulation for the questions), but Ewan Hoyle, the founder of LDDPR, has highlighted the importance of asking the right questions when trying to establish the public’s views on drug policy reform.

Interestingly, his conclusion – that when asked to choose between various regulatory options, the British people are comfortable with strict control and regulation as a solution to our drugs problem’ – also seems to apply to the American people, too.

An Economist-YouGov poll carried out last month asked respondents to agree or disagree with the statement ‘Marijuana should be legalised, taxed and regulated’. In every age group polled – even over-65s – more people favoured than opposed legalisation. Although it is hard to say definitively, it seems reasonable to suggest that this is due to the mention of the more public-friendly measures of taxation and regulation – measures which, although they go hand in hand with legalisation, are so often omitted in opinion polls on drug policy reform.

It certainly seems to be the case that this question, which spells out a little more clearly what is actually being advocated, delivers more positive results than the more traditional ‘Legalisation of marijuana: yes/no?’ – although even when this latter question is asked we can observe a long-term trend of growing support for change in the US:

The above graph, compiled by polling guru Nate Silver, amalgamates all the data he could find on the issue of the legalisation of cannabis/marijuana – including the results of several Gallup polls and the General Social Survey – and shows that there has been an upward trend in favour of such a move since 1990.

Consequently, while especially encouraging findings emerge from polls that provide a more comprehensive overview of what drug law reform might actually entail, even the findings of polls that do not give a sufficient account of the options for reform show a sustained, long-term increase in support for more progressive drug control measures.

The story with regard to cannabis was roughly similar in the UK until the mid-2000s, since when support has dropped from between 40-50% (up from 15% in the 1980s), to between 30-40% today. Speculating as to why this has happened is for another blog, although it probably reflects the shifts in the nature of the UK cannabis market towards stronger varieties and the associated health concerns raised, or arguably hyped, by the media and politicians during the interminable cannabis re-re-classification saga. It may also reflect the fall in use noted by the British Crime Survey over the same period.

Hopefully the growing support more broadly – in the UK and elsewhere – will reassure policymakers and politicians that the legal regulation of currently illegal drugs is not only practical but politically possible. Remember that the steady change in public opinion has happened despite, until very recently, near universal political and media hostility. If any major political forces got behind reform it seems likely that things would change much more quickly – it's a question of who wants to pick up the baton and claim the plaudits.


Jake said...

I would love to know the general public's true perception of 'drug use' via a well-worded poll. One stumbling block people seem to have when I talk to them about legal regulation is that 'use will increase' i.e. we shouldn't enact pragmatic, evidence-based policy because initial use 'may' increase, even after I explain that said use would be safer and the other benefits of regulation.

I wonder if something like this would be a good question:

'If legal regulation of all drugs were to occur today with a 5% increase in use over the 3 years following, then decrease to -5% of current levels thereafter with associated reduced levels of crime, violence and addiction, would you support legal regulation?'

I'm by no means a pollster, but I think peoples concerns (i.e. potential rise in use) regarding legal regulation should be addressed alongside decriminalisation/legalisation/prohibition polls, see if they would be willing to support a policy that could have 'negative' initial outcomes in their eyes - higher use, as some people are not prepared to make that 'sacrifice' for an overall long-term benefit. See how altruistic and compassionate people really are...

Eddiesilence said...

Let's reframe the question about how to control or regulate drugs by putting people in the centre. The flipside is the key: regulate/legalise/control drugs - or else what? Currently, it's or else prison.

Would it not be clearer to gauge society's view from a survey question framed thus:

Should we send people to prison for having/trading/producing, etc.; cannabis/opium/LSD/tobacco/alcohol etc.? If yes, then for how long, and to solve what problem?

HomeGrownOutlaw said...

Great piece, one in which I fully agree.

The case in point that I tend to point people towards is the Lib Dem poll vs Professor Nutt's; Estimating Drug Harms A Risky Business:

You'll quickly notice, in Estimating Drug Harms, a clear consensus is in favour of law sending a message, for harsher penalties, but without actually having the judicial reprisals. The questions posed were based around "legal, illegal" terminology.

The Lib Dem poll of course focuses on prohibition and regulation. This use of language drew different results.

I fully maintain that to change the answers is to change the question.

Brilliant piece once more!

Ewan Hoyle said...

It's interesting that the big difference between the Zogby poll that found 44% support and the Economist poll that got 58% was that the Economist reminded people that legal like alcohol and tobacco means illegal to minors.

HomeGrownOutlaw said...

Ewan, there's almost a slogan & campaign to be found in that. Interesting outlook.

Recovering Ex-User said...

As an ageing researcher, I can tell you that many things influence people's responses to surveys, not just the wording of the question and the multiple-choice responses provided. These include who/which agency is asking the question, how anonymous and confidential the survey is, how long the survey takes, whether payment or costs are involved, what the time and date is, where the survey is carried out, and how intoxicated the respondent is. And that's just scratching the surface. For one big instance, how anti-drug society and the mass media are at the time of the survey has a huge impact on the honest of responses about use of drugs. Few surveys include proper checks on validity and reliability - even the British Crime Survey has no measure of false denials. Our research on the prevalence of drug use is one huge ugly mess.

P. Dantic said...

A good survey question is specific, simple, direct, precise, unambiguous, clear, relevant, fair, relevant and realistic. In short, good survey questions are rare. And when it comes to drugs research, good questions about people'e use of drugs and their attitudes to them are extremely rare. And the second part of such interrogative research methods is the response format. Rating scales are more useful than YES/NO response formats, but since people can generally not make more than 10 distinctions on things like agreement or frequency, then researchers should rarely exceed five or seven points on the rating scale. In short, our national surveys of drug knowledge, attitudes and behaviour are creatively retarded and poor (CRAP).

Dee Empty said...

My favourite daft survey question, apparently designed to improve on the question 'have you ever been offered drugs?', is 'have you ever been in situations where drugs are available?'. If the response option was available, I would always ticks 'yes, I live on the planet Earth'.

Tom said...

Not really sure the policy makers will pay any attention at all. In the U.K. we have gone from Class C to Class B and the new Auton ruling, which means most if not all Cultivation/Production cases are going straight to Crown Court, with a custodial recommended for anything over 4 plants.
The cost of this new ruling may prove to be the straw that broke the Camels back due to the far higher costs involved to the tax payer.

Anonymous said...

It is really easy to skew statistics of all kinds. And even easier to sway the public by having a single person promote both sides of the story. There are so many politicians and surveys out there asking similar if not the exact same question that I feel like a survey monkey.