Taking Science Seriously
If there is any knowledge or understanding among researchers that is scarce, it is the understanding for research. What I mean is not that researchers fail to use their methods correctly, that they fail to properly test for whatever biases, or that they fail to consider missing/omitted variables. No, there is a lack of understanding for what science is and can do.
The key word here is “understanding,” which I mean in a quite Weberian sense.
Research is so specialized that it is more about technique than about actual thinking. Sure, a lot of thinking goes into figuring out alternative hypotheses, finding the correct proxies, and so on. There is also some heavy thinking involved in figuring out how to interpret the results and to connect them to a theoretical framework.
See the error here? The common denominator? No?
Unfortunately, I’m willing to wager that this is the case among researchers as well (though I have no statistics to back it up). It is because research has become the art of quantitative, statistical analysis – it is not about understanding stuff. The common denominator in the paragraph above is exactly that: quantitative statistics. That’s taken for granted as the best (sorry, the only) way of getting any knowledge out of any research project. Based on this (in my view rather unfounded) assumption, a lot of thinking is going into figuring out alternative hypotheses, finding and arguing for proxies, making assumptions about the dataset that at least appear reasonable, etc.
When the statistical analysis is done, it is connected to some theory that happens to seem to fit to the results. Sometimes theory is the starting point, but usually the study’s data and statistical analysis causes a need to change the theoretical perspective. There’s reason for this: data is what goes as science, and the more you have the better is your scientific contribution. It really boils down to the fact that a bigger data set is better than a small data set simply because as “n” (the number of observations in the data analyzed) increases, so does the statistical significance.
Really, I’ve been to academic seminars where presenters proudly declare that “this is what’s so great with huge datasets” – because it doesn’t matter what hypothesis tests they run, all of them will come out as “significant.”
So what does “significant” mean? It means that the results are reliable in the sense that they are, considering the null hypothesis, within a certain (rather arbitrary) percentage of assumed cases that are sufficiently (also pretty arbitrary) non-zero. The reason a sample, since we usually never study populations, needs to be random is that it is the only way our assumptions about the data (which by the way usually includes the bell curve distribution, even though we know that’s a bit of a stretch) can hold.
A lot of thought has gone into the statistical methods, which of course are improved all the time. Some of the assumptions are almost outrageous, but they seem to work most of the time. Of course, there are also other assumptions on which the very use of the statistical methods rely. Such as that science must be inductive (not deductive), that what is observed is what it is (and not affected by the observer), that what is measured is how it is measured, and so on.
Graduate school is about learning one’s trade of choice, which means learning the techniques used so that one can be a card-carrying member and run the same types of tests. The understanding for the philosophy of science among scholars is for this reason severely lacking – the ontology is usually simply asserted, while the epistemology is commonly ignored. This is part of the reason we’re relying on quantitative statistical “analyses” of social phenomena which are neither measurable nor objective.
The question we should ask is what we can learn from using a bunch of (usually rather bad) proxies for subjective assessments about a structure or pattern in actions taken by socially embedded individuals who have experiences and continuously learn. We don’t know what they think, what they perceive, how they value, what they value, what they try to achieve, or what influenced that particular action – or how these individuals’ assessments are interrelated. Yet we collect observed, objective “data” and run statistical regressions to find out where, for instance, we can fit a straight line that minimizes the squared distances from observations that aren’t actually on that line.
An interesting example here is what’s considered “normal” in medicine, even though we’re now taking a step out of the social sciences (where the problem of statistical analysis is rather obvious). That word, normal, which is the benchmark for most assessments of our health is commonly the average or median or mode of people who have undergone a specific test (like counting cholesterol particles in blood). Let’s look at two fundamental questions:
What is the relationship between “normal” and the population as a whole? We don’t know. Most tests – our data – are done on people seeking care for one or another reason, and are ordered by physicians. They are ordered because they seem reasonably relevant for whatever health issue the patient seeks care. In some cases there are control groups in clinical testing that neither have issues nor take medication, but even where this is the case the abundant majority of observations is from people who have some symptom that can be related that which is tested. “Normal” is then based on this data, perhaps with statistical “adjustments” to account for this and other problems – and normalized using the bell curve and whatnot.
What is the relationship between “normal” and optimal health? Even if the previous question can be answered by affirming that “normal is the average/median/mode of the whole population,” we cannot say whether this type of “normality” is in fact good health. On a scale from poor health to good health, why would we want to be in the middle and not in the “extreme”? If we take the American population as an example, even with data from some 350 million people – how do we know that the population is normally distributed around a mean that is in fact good or optimal health? We don’t. It appears rather likely that the whole population fits squarely in one of the “tails” due to cultural and traditional aspects of nutrition and cuisine. Americans eat burgers and fries, pizza, and whatever. That’s not necessarily food that makes the population normally distributed around optimal health.
The fact is, we don’t know what optimal health is. It is probably not grains, processed foods with high sugar or high-fructose corn syrup content, and so on. There is an argument to be made for nutrition aligned with optimal health being something very different than agricultural products (like the primal diet focused on lots of fat, meat, and veggies). So even if we use a cross section of the whole population – even the world population – we don’t get an average or median or mode that is necessarily close to optimal health. We simply don’t know, and this should be an important aspect when studying health. Is it? Not really.
Take this recent article about how one should probably throw all of one’s vitamins in the trash as an example. It seems to only rely on statistical studies of people’s health, which should raise a few flags. Then there are implicit assumptions about normality, which are not even stated. And then that there is an optimal level of all those measurable levels (rather than optimal levels depending on how one’s body works, what one does during a normal day or in one’s life, etc.). And it also seems we’re not even considering the quality of supplements but only “supplements.”
(Also, I’ve learned, studies on vitamins usually test very low levels of supplemental vitamins in order not to do harm, which also makes results questionable – what if much higher doses are needed? Why is a low-dose supplement necessarily healthier than a high-dose?)
The claim is that “the data increasingly suggests that most people simply do not benefit from supplements.” Maybe that’s so, but to say anything about any data and what they “suggest” we must know something about the benchmark. And the benchmark is mysteriously missing from the article (as well as, one can confidently assume, the studies themselves). Instead, we get comparison of different types of studies. As the authors says, there are reportedly “large effects in observational data, nothing in randomized trials.”
OK, fine. But effects on what exactly? On the frequency of “cancer or heart disease.” But the definition of health, and especially optimal health, is not simply “no cancer or heart disease” – it is much more than that. It should at least include energy levels, mood (especially mood swings), strength, lack of disease and pains and aches, ability to heal and recover, and so on. Even if one has a deficiency of vitamin D3 that weighs heavily on one’s health, it may be the case that there are other deficiencies as well – so maybe adding only that vitamin doesn’t solve “cancer or heart disease.” It is too simplistic.
And even if there is a relationship between vitamin D3 deficiency and cancer, a bunch of recent studies indicate that cancer tends to become an issue in people with poor immune system – probably where there is poor nutrition through decades of reliance on toxic and nutrition-deficient foods. Consider this sort of contrasting conclusion from the article:
To be clear: Serious vitamin deficiencies can cause serious problems (scurvy in the case of vitamin C, rickets in the case of vitamin D, beriberi for vitamin B). But if you live in the developed world and eat a normal diet — even a pretty unhealthy one — you will be nowhere near this kind of deficiency.
Yes, let’s be clear. “Normal” levels of vitamin D, which is produced by the body as sunshine hits the skin, is based on a population that is always clothed, almost always indoors, and that has been told to always use SPF not to get skin cancer. This is not in any sense “normal” considering how the human body developed over hundreds of thousands of years of evolution, so how can an average or median or mode value in any sense be considered “normal”? Probably because it doesn’t cause rickets. But there is no reason to assume no-rickets is the definition of health. It certainly isn’t optimal. (On the contrary, there is reason to believe that most of us are vitamin D deficient and suffer from it even if we maintain levels high enough to avoid rickets.)
But let’s not get lost in the health issue and the revolution that is happening right in front of our noses thanks to movements such as paleo, primal, raw, etc challenging traditional [Western] medicine. What is of interest here is the rather outrageously lacking analysis used in science in general – by force-feeding studies with “data” analysis and relying on aggregates and quantitative measures and statistics – and that is all too obvious in the article on vitamins.
It all seems clear-cut: there is no “evidence,” so let’s just throw out those vitamins. But then, when one thinks about it in a structured way and attempts to figure out what is actually going on – it just seems to be hot air. There is absolutely nothing there. And that’s why we are prompted to “don’t take your vitamins.”
Structured, rational thinking about community, cooperation, and the market