How to p

According to a recent “study”, there appears to be a “correlation” between locations of ‘Mad Cow Disease’ outbreaks in the United Kingdom in 1992 and areas that voted in favor of leaving the European Union during the referendum. So, yeah, Mad Cow disease leads to Brexit. Of course, the whole thing is a hoax; there is no correlation that we’re aware of, the images are fake, but if nothing else, it’s mostly funny (if it weren’t for the Bad Things like this that went along with the Brexit).

So why bring this up for reasons other than the opportunity to use a snarky title and a lot of tags that make no sense when put together (13 tags – WordPress isn’t going to like this) And why are there tomatoes in the featured image? I’ll get to the tomatoes later, but let’s face it: we humans have this need to know why, ever since we were children pestering our parents. As we grew up, it turns out that most things in life have more than one reason for coming to be, and they can get really complicated. So we turn to good old Occam’s razor: if there’s many explanations, choose the one with the least assumptions (or as many have learned it, the easiest explanation is the best one). So we want an easy explanation for the complicated things. Sadly, that’s where some really shady statistics can come in.

A lot shady statistics practices come of relying too much on a “p-value”, which has to do with the likelihood of finding an extreme result (here’s a quick read). The lower the value (especially if the value is less than 0.05, 0r 5%), the more extreme it is, and you can make a stronger claim for whatever hypothesis you’re trying to make. For example, Fivethirtyeight.com did a survey earlier this year regarding nutrition using a lot of seemingly disjointed variables including various types of food, pets, religion, and SAT scores. John Oliver also covered this on Last Week Tonight. Did I say ‘seemingly’ disjointed? What I should have said was intentionally disjointed. According to their entry on github, having huge data sets with many, many variables means it’s “easy to p-hack your way to sexy (and false) results.” That’s why according to their survey, eating raw tomatoes is linked to practicing Judaism (p < 0.0001), drinking soda is linked to getting a ‘weird’ rash in the past year (p = 0.0002) , drinking coffee is linked to cat ownership (p = 0.0016), and eating table salt is linked to a ‘positive relationship with Internet service provider’ (p = 0.0014).

Of course those results don’t make sense. They’re not supposed to. I drink coffee and own cats, but that’s anecdotal evidence (and one heck of a coincidence that a lot of people are like this). Also, there’s no theory in any form of research behind salt consumption and your relationship with your ISP unless you’re a gamer suffering from tons of lag, in which case forget eating salt – you’re already salty (and blessings be unto those who have never experienced lag). But that’s the dangerous thing about p-hacking or data mining; you can get the numbers to say whatever you want even without creating false data. That’s right – the data is not falsified in any way, but it comes from essentially deceptive practices. Two things can come from this: 1) you can ‘propose a hypothesis’ once you’ve found a significant (you found a relationship that’s got a very low p value) relationship instead of the other way around as it is with real science, and 2) you trust the numbers too much, and it can’t even be demonstrated in real life.

The second point is one of the biggest abuse concerns of social scientists and statisticians, often as a ‘type I error’, or the false positive. The math says it should be true, but reality has other ideas. Remember the very significant link between eating raw tomatoes and Judaism? According to a report by the Jewish People Policy Institute, the countries with the largest Jewish populations are Israel and the United States. Now, what are the countries with the largest tomato consumption? If we’re talking straight weight, it’s China and India. If we’re talking by person, it’s Libya and Turkey (you were probably thinking Italy – yeah, me too). So there’s your false positive set up from a shaky hypothesis and shady practices; the silver lining is that Fivethirtyeight’s practice was intentionally flawed, and they never proposed a hypothesis regarding tomatoes and Judaism. There are dangerous consequences for misusing science and stats.

So what does this mean for us non-statistics, non-social science research citizens? Pretty much one thing: if there’s some really shiny, sexy science with results too good to be true and it makes the morning news, we need to be able to question it. Some people are much more skeptical of these things, and they’ve got every right to be – it just makes it look like science is ‘selling out’ its values just to get page views or funds. Question where the study came from (who did it, who reported it), how they did it, who funded it, and so on – there may be more to it than face value. There’s very little in life that only ever has one reason for being the way it is.

Personally, I really don’t want to get extra ads from internet service providers trying to make me not dislike them just because I’m Asian (p = 0.03) and I regularly have cooked oatmeal for breakfast (p = 0.004).

postscript: I post on Mondays and Thursdays, and this just happens to come out on the 4th of July, Independence Day here in the United States. This post has nothing to do with it, but have a Happy Independence Day to my American readers, wherever in the world you may be.

postscript

Featured image credit: Wikimedia Commons (CC BY 3.0, user: Goldlocki)

Advertisements

10 Comments

  1. Wow….I am so glad you commented on my post, thereby coming to my attention….this post (and you, of course) is so engaging and interesting…..and delightful and entertaining. I am a huge fan of your writing style. And, despite the topic (I only had to take Stats for one semester in order to complete my Bachelor of Social Work degree–but the dread factor is legendary), this felt like a breath of fresh air. And, I understood every word of what you were saying! You sure do have a gift, and I am really looking forward to reading much more. I need to get back to sleep, but will visit you soon….after work, I imagine. Thank you for sharing 🙂

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s