Latest posts by Amber Hathaway (see all)
- Tips for Debunking Discriminatory Science - October 20, 2017
- What the Afghan Girls’ Robotics Visa Issue Suggests about Trump’s Muslim Ban - August 1, 2017
- Summer Reading for Feminist Horror Fans: Woman in White - July 2, 2017
In August, the scientific journal Nature published an editorial arguing that science does not support or justify discrimination. This fact should go without saying, but unfortunately it’s relatively easy for unscrupulous individuals to manipulate data to support their agendas. If you don’t have formal training in statistics, it can be hard to sort out science from the sea of claims posing as science. Here are some tips for figuring out whether a claim has merit.
Consider the source. Many of us are aware that sites like Fox News and Breitbart are peddling a conservative agenda, so you can expect their interpretations of the data to be molded to fit their ideology. However, there are lots of less well known sites and blogs that are similarly skewed, so watch out for cues in the tone of the article suggesting that the author is basing their interpretation of the data on their ideological viewpoint.
Be wary of clickbait. In addition to skewing to fit an ideological agenda, some articles may be worded to generate clicks and shares. Unfortunately, clickbait is very profitable, so writers will use evocative titles that may not reflect the scientists’ intent to lure in readers. As the 2014 NPR April Fools joke shows, too often people draw their conclusions from the article’s title alone. Be sure to read the article to see if it actually supports what the title claims it does.
Another thing to look for is whether the science presented in the article supports the claim the article is making. When reading the article, look for direct quotes from the researchers and see if their statements are as forceful as the rest of the article. Suppose an article claims that gender differences in SAT scores prove that women are bad at math (I’d rather not link to one, but there are plenty of articles out there making this claim). The article quotes a researcher as saying, “we observed a statistically significant difference between the mean SAT mathematics scores of male and female test takers.” The researcher in this hypothetical article is not making any claim about men being mathematically superior, so be wary of the author’s conclusion (more on “significant” differences soon).
It’s worth remembering that no researcher is truly objective. We all have our reasons for wanting to study the things that we do and often a researcher will have a particular result that they’re hoping to find. If a researcher is expecting to find a difference but finds none, they may choose not to publish their result. There is also a small but vocal population of researchers who are openly trying to prove that certain people are superior in some way to others. If it seems like a researcher is pushing a discriminatory angle, you may want to look at their other research and see if there is a general trend of promoting discriminatory views.
There are also inadvertent misrepresentations of data that arise because the writer does not have the scientific background to interpret and convey the research. For example, in statistical jargon a “significant difference” means that it’s probable that the difference exists. It does not mean that the difference is large or important. In fact, the difference could be very small, but someone who is not familiar with statistical terminology might assume that a “significant difference” is a big difference.
It’s also important to understand that when researchers are talking about differences between two populations, they’re usually talking about differences on average. If these differences are small, then that means there will be a lot of overlap between the two groups. Take, for example, SAT math scores. If you look at the mean SAT math score for girls and the mean score for boys, you will find a small difference between the two averages. However, if you look at individual girls and individual boys, you’ll find that many of them share the same SAT math score. You’ll find girls with perfect scores and boys with low scores. The difference in averages doesn’t tell us much about how an individual girl will perform relative to an individual boy, and it certainly doesn’t prove that boys are mathematically superior. On almost all measures, there is much greater difference between members of the same gender than there are between genders, and the same can be said for race, class, sexuality, ability, etc.
Another important caveat is that finding a statistical difference does not tell you anything about the cause of that difference. You may have heard the refrain that correlation does not imply causation. What that means is that just because two things are related statistically does not mean that one causes the other. There are plenty of articles out there claiming that the gender gap on the SAT math section proves that men are biologically better at math than women. We’ve already seen one problem with this assertion, but there is another serious issue as well. It’s an enormous and unfounded leap of logic to say that biology causes this difference because there are other factors that could be responsible at play, the main one being socialization. If you wanted to examine whether gender differences are biological in origin, you would need a group of boys and a group of girls who have been socialized in exactly the same way, a virtual impossibility in contemporary society. There are reasons to believe that socialization affects educational outcomes, so jumping to the conclusion that observed differences are biological in origin isn’t scientifically sound.
These are just a few things to be mindful of.
For more on issues relating to research on gender differences, I recommend Cordelia Fine’s Delusions of Gender.
For more on how statistics can be manipulated, How to Lie with Statistics provides a good overview.