Consume Food Science Wisely - Knowledge is Power
‘’You can go back to eating butter’’, claimed a New York Times article published in 2014. The article titled ‘Butter is Back’ cited a study that conducted a systematic review of evidence about associations between fatty acids and coronary disease, and concluded that replacing poly-unsaturated fats with natural saturated fats had no substantial impact on cardiovascular risk. In one fell swoop this headline rubbished the decades of belief that polyunsaturated fats such as hydrogenated plant oils were great for us. Another broad, sweeping statement made headlines only last month when an article arrived at the misleading conclusion that ‘’scientists find no benefits to time restricted eating’’.
Although the field of nutritional studies is brimming with conflicting information, what remains consistent across the body of literature is the extreme nature of claims that most studies make. Veganuary, The Starch Solution, The Whole 30 Diet - these are only a few examples of diets that are selling, in one way or another, the promise of a healthier, longer life.
It’s a far too familiar story - eggs, wine, coffee, fish, fats, red meat, carbs, Mediterranean diet, vegan diet - there’s at least one observational study in the field of nutritional research implying a causative link of the aforementioned to heart disease, longevity, obesity, mortality - you name it. Under the garb of statements like ‘’data suggests..’’, findings from observational studies on nutrition trickle down to widely read newspaper columns. The results of these studies not only create sensationalist headlines in newspapers, but are conclusions that are routinely considered by physicians, and passed off as recommendations to patients, and even form the basis for several public health policy decisions. In 2019, Mayor Bill de Blasio announced that all New York City public schools would enforce ‘Meatless Mondays’ as a move that was meant to be ‘’good for the kids’ health’’.
A lot of these claims, conclusions and perceived benefits, however, are no more than misleading messages, and remain unsubstantiated without the backing of rigorous research methodology. In fact the reason why there is so much conflicting advice around nutrition is because there are serious design and data analysis issues unique to the field of nutritional epidemiology. Some aspects of the field are so pervasively and irreparably flawed that the best thing we can do about most findings, ‘’promising’’ research and papers in the field is to look out for certain red flags (that we will discuss below), and simply ignore most of the literature on nutrition science.
Before examining the often misleading, exaggerated and even flat-out wrong conclusions of a given study, we must begin by scrutinizing the very questions that researchers set out to study. Questions in nutrition science- much like any other research field - are swayed by intellectual and financial conflicts of interest.
When rejection rates for renowned scientific journals are already above 90 percent, it’s difficult to not set out by studying and trying to prove eye-catching theories. What's even worse - getting a null result with one’s initial research question, and then engaging in "p-hacking" - a mechanism where one looks backwards in the data and makes adjustments to find associations. This is called "statistical fishing", and is all too common in the field of nutrition research. When researchers are operating under the immense pressure of ‘’publish or perish’’, the questions need to be eye-catching, and the results need to look promising, or better yet - controversial. Take for example, this study that came to the conclusion that being overweight lowers the risk of all-cause mortality!
You don’t need to dig deep to find spurious correlations in the most bizarre and unconnected things. Here is a favorite site of ours by Tyler Vigen, a data geek who tries to educate with humor that you must not just believe folk who sell stories and products with headlines. How about this one based on official public health data: “New data shows dramatic link between eating cheese at night and dying in your sleep.”
Much of the methodology in nutritional research - from data collection to its analysis - remains notoriously unscientific, mostly based on observations of a small number of cases. Most nutritional studies rely on gathering their data by simply asking people what they ate, and expecting them to report accurately. Few studies conduct direct observation and measure exactly what people ate during the course of the study, which of course would be a more expensive and difficult study design to execute.
In 2013, a study aimed to put a number on how far off the reported data on diets are from what people actually consume. The study looked at data from four decades of the biggest US government database on diet - the National Health and Nutrition Examination Survey (NHANES) and found that, ‘’Across the 39-year history of the NHANES, i.e, data on energy intake/calories consumed by individuals, the majority of respondents were not physiologically plausible. That is, if people ate what they said they ate, they would have starved’’. This shows that more than half the reported data is demonstrably inaccurate - and this is only accounting for data that was under reported.
Further, nutritional studies rarely go on for long enough to track the decades-long course of several diseases and untimely death. In other words, the studies aren’t directly studying the effect of X on life-span, but rather on easily measurable health markers like cholesterol levels, blood pressure, blood sugar levels etc. which are then used to make a case for correlation with life span. The issue with these arguments is that meta-experts have shown that changes in these markers often don’t correlate, and rarely show causal relationship with long-term health as we are often led to believe. In fact, when studies have gone on for long enough, they often undermine the status quo understanding in the field of nutritional epidemiology. For example, the longest study of overweight individuals wasn’t able to convincingly show that overweight people are likely to die sooner - and that is going against pretty much the most agreed upon ‘’fact’’ in nutrition research. The truth, given the structural flaws in study design and analysis, is that there can be no agreed upon ‘’fact’’ in this field.
Randomized controlled trials (RCTs) are seen as a panacea to nutritional studies, and a more rigorous alternative to observational research. RCTs compare how one group responds to a treatment against how an identical group performs without the treatment. Dr. John Ioannidis from the Meta-Research Innovation Centre at Stanford has found that ‘’80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials.’’ Moreover, with individuals’ biology and biome being so unique and changeable can we really ever control well enough for an RCT as to how an individual’s results can be reliably mapped or scaled at a population level? Even twins with identical biology will have different glucose and insulin levels after consuming exactly the same food - gut microbiota or ‘micro-biome’ being the best variable to adapt.
The exposure of interest in nutritional epidemiology—dietary intake—is complex, with interactions and synergies across different dietary components, which can be difficult to study with the use of a linear drug trial approach. Dietary intake is not a single variable but a set of several intercorrelated continuous variables, because nutritional variables are all correlated with each other. In other words, if a single nutritional variable is causally related to any health outcome, several others will yield statistically significant association if the data set is large enough. Considering that most research now involves big data, it is likely that almost all nutritional variables are associated with almost all outcomes. Further dietary intake varies from person to person by eating behavior, social and cultural factors, pre-existing disease etc. Not to mention the variation in age, metabolic profile, genetic background, the environment in which they live - these are all entangled in the potential influence on health outcomes of a single dietary component. Not only do dietary studies not address the issue of confounding variables in nutritional studies, but also do not acknowledge that it is almost impossible to come to a conclusion with an acceptable level of correlation between a single nutrient/food group/food item using a simple questionnaire.
It must be noted though that observational studies as a methodology isn’t the problem, its use in nutritional research to imply causation is. Despite its limitations, there are several well accepted causal inferences that are supported fully or partially by observational studies. For instance, associations between hypertension and stroke, smoking and lung cancer, LDL and HDL cholesterol concentrations and risk of ischemic heart disease. Observational studies conducted on smokers to study the association with lung cancer shows that there is a 1000% higher risk of developing lung cancer for smokers as compared to non-smokers. In the UK Million Women Study where the median age was reported at 55 years, women who quit smoking completely at ages 25–34 or 35–44 years had only 3% and 10% of the excess mortality, respectively, seen among women who were continuing smokers. Relative to this, the correlational odds ratio between consuming red meat and developing colon cancer is 1.1 to 1.2 - virtually nothing. This is because observational research study can never be interpreted in an information vacuum. The strength of the evidence needs to be weighed against potential bias in study design, and bolstered by existing evidence collected on the same subject by other methodologies.
It is clear that nutrition studies are laden with epistemological problems. Over-reliance on observational studies, suboptimal self reporting, challenges detecting tiny effect sizes reliably for nutritional risk factors, accounting for sizable confounding amongst many nutrients and other variables, and the inability to measure diet accurately are only some of the issues in nutritional epidemiology - using which researchers are able to make broad, sweeping statements about etiological questions.
It’s very easy to be swayed by eye-catching - sometimes jaw dropping - results of medical research, but it is important to understand that much like any other field of research, nutritional studies are pervasively flawed. A study reviewed over 50 common ingredients from random recipes and found that 80% of the ingredients were correlated with cancer risk across nutrition science literature. To believe most of these findings would evidently mean to starve! So we suggest three things whenever you read about the ‘‘latest breakthrough’’ in nutrition studies:
First, never just believe or act on anything you read without proper scrutiny.
Next, if you don’t have the time to scrutinize information yourself then ask a group of inquisitive optimists, like ourselves, or use Examine.com - they are the best independent group out there today scrutinizing diets and supplements.
Finally, and most importantly, remember that you’re an individual, and that even your genetic twin will not have the same response to a diet as you.
And remember that whatever ethically-farmed, nutritionally-diverse, whole-foods you’re eating, doing so slowly, socially and sitting down, is just as critical for happiness and longevity - now, that is not just another headline!