How to read scientific studies

Newspapers love scientific studies. It gives them the opportunity to promote big bold headlines. Usually telling us that something is about to kill us and the world is on the verge of disaster.

The Daily Mail in the UK is the master of such scare stories. Some recent gems have recently informed us that eating red meat will kill you, brazil nuts will give you a heart attack, oral sex will give you cancer, morphine will give you cancer, mobile phones will give you cancer, anti-ageing creams will give you… you get the idea.

Now I’m a gym guy and my focus is on health and fitness. Nonetheless, nutrition plays a huge role in getting results in the gym and I am constantly battling concerns from clients who have read the latest scare story and are worried.

One way I try to counter misinformation, is to ask people to look at the actual study from which the newspaper is reporting. How were these studies conducted? Are they credible? What did the researchers conclude and is it the same conclusion drawn by the journalist who wrote the article? I want to take the red meat will kill you story as an example, as this is a particular bugbear of mine. According to the Daily Mail “eating red meat is not much healthier than drinking arsenic”, a rather extreme premise, but is there any truth behind it?

One of the key tests of the legitimacy of any study is how many people participated and over what time period. A study of 100,000 people over 2 years will provide much more useful data than a study of 5 people over 2 days. In this case 120,000 people participated over 30 years, so all seems good there.

The next point is how data was tracked, and here we hit our first problem. The study used a food frequency questionnaire which was updated every 4 years. Essentially the main question was ‘how often do you eat meat?’, with answers ranging from never to 6 or more times per day.

Just think about that for a second. If I asked you right now how often you had eaten meat in the past 6 months (let alone the last 4 years), would you be able to give a reply that is remotely accurate? Can you even remember what you had for lunch 3 weeks ago?

So essentially our starting point is a very, very rough idea about peoples’ eating habits, perhaps not the type of hard information from which strong conclusions can be drawn.

The second, massive red flag of this study is the link between correlation and causation and something which is seen time and time again in the interpretation of medical studies.

Imagine that you received a visit from a Martian, who you graciously decided to give a tour of the earth. You finish the tour and the Martian tells you that he loves the planet, but he would definitely advise ridding the world of the police. That would obviously seem strange as you know that the police are responsible for law and order, but the Martian sees it differently so you ask why.

Simple, he says, the police were at every crime scene we passed so they are clearly responsbile for all criminal activity. This is an example of correlation, not causation. Yes the police were present at the scene, but they were not responsible for committing the crimes. You can see why the Martian would draw this conclusion, but it is obviously false.

Drawing correlations in studies is a very dangerous business indeed, because there is no proof that causation equals correlation. In our study, we see that the group who had a higher consumption of red meat also tended to be smokers, overweight and drank more alcohol. Could it be possible that these were the cause of increased mortality rather than red meat intake?

Now I’m not saying definititively that red meat won’t kill you (although I suspect you will all be just fine), but the point is that I don’t know and neither do the people that conducted this study. They are simply drawing very tentative correlations from the (admittedly poor) data they have at their disposal. The journalists have then taken these tentative correlations and turned them into scaremongering headlines, so suddenly we are all about to drop dead thanks to the steak we had for dinner last night.

So beware of big, scary headlines from newspapers whose sole purpose is to sell as many issues as possible, do some research and draw your own conclusions.