Thirty years ago, the sociologist Craig Reinarman observed that there's something "woven into the very fabric of American culture" that makes us susceptible to believing that a "chemical boogeyman" is to blame for "society's ills." He added that every moral panic about drugs since the 19th century has been fueled by "media magnification" in which the danger of a particular substance is dramatized and distorted.
Now that recreational marijuana is legal in about half of U.S. states, and more Americans are consuming weed than ever before, the chemical bogeyman is back, and he's armed with a new paper in the Journal of the American Heart Association by researchers from Harvard and the University of California, San Francisco.
This study, which was amplified in The New York Times and The Washington Post, commits so many egregious statistical errors that it's a poster child for junk science. The paper would be comical if it didn't offer bad medical advice. The researchers did almost everything wrong.
Which is not to say that the authors committed fraud or misconduct. In fact, they did exactly what Ph.D. students are taught to do, what journal editors look for, what referees approve, what universities reward, and what granting agencies fund. Because the paper uses conventional methods to arrive at false conclusions, it speaks to the profound crisis in academic research.
We've forgotten that the point of scientific studies isn't seeking the approval of institutions. It's the pursuit of truth.
The study's setup is simple enough. The U.S. Centers for Disease Control and Prevention runs an annual telephone survey to ask people about their health and behavior. The authors looked at rates of three cardiovascular problems in users and nonusers of marijuana.
It's an observational study, not a controlled experiment, which means you can't infer causality. Most researchers are transparent about the limitations of their data.
Not in this case. The authors went so far as to derive a "clinical implication" from their finding, writing that patients should be "advised to avoid smoking cannabis to reduce their risk of premature cardiovascular disease and cardiac events." The media took them at their word.
This is quackery. An observational study simply can't show that quitting marijuana reduces cardiovascular problems.
Observational studies can only show correlation, not causation. And in this case, we can be certain there is no causation because the phone survey measured marijuana use that occurred after the onset of disease. It asked participants about their marijuana use over the last 30 days and about their cardiovascular problems at any point in the past.
How could a bong hit a week ago, or a THC gummy last Saturday, possibly cause someone to have a stroke 10 years ago?
Another problem is that the data don't show the people who had cardiovascular problems; they show the people who survived cardiovascular problems. Let's say hypothetically that marijuana reduced cardiovascular mortality.
We'd expect the study to show more marijuana users with heart disease and stroke because they would be alive to report their cardiovascular problems in the phone survey. More of the nonusers who had cardiovascular problems would be too dead to respond.
This is for the same reason that people who wear motorcycle helmets are hospitalized more often for accidents—because three times as many non–helmet wearers die before they get to the hospital.
Though the data come from phone surveys, the authors make no allowance for the unreliability of what people might tell a stranger on the phone about their health and drug habits.
The researchers even had the gall to assert that surveys of this kind tend to be accurate, while referencing only two studies in support of that claim—a 1982 paper that found that such data are "so unreliable" as to be almost impossible to interpret and a 2004 study showing that about half of self-reported cardiovascular problems were fictitious.
Worse, the fictitious self-reports weren't random, but were much higher for some groups than others, which would distort any findings.
The researchers also had to deal with the other behaviors that correlate with marijuana use and are known to impact cardiovascular health. Marijuana users were more likely to be current and former tobacco smokers, more were men, and they drank more, all of which you'd expect to correlate with cardiovascular problems.
In fact, marijuana users in the study had fewer of all three types of cardiovascular problems measured in the study—an inconvenient point that the authors only mention in one jargony sentence late in the paper.
Coronary heart disease, for example, afflicted more nonusers of marijuana than either daily or nondaily users.
This doesn't necessarily negate the authors' thesis. To drill down into the data, they should have tried to find a meaningful subgroup—like overweight middle-aged men—in which people who used marijuana had more cardiovascular problems. Then they could ask whether marijuana, or a different factor, explained the difference.
So how did the researchers claim to back up their thesis? They tortured the data until they offered a false confession.
The main tool they used was logistic regression. The authors examined the association of marijuana use with strokes and educational attainment.
Here was the rate of strokes for users and nonusers of marijuana based on time spent in school.
As you can see, more education was correlated with fewer strokes. But for each level of education, marijuana users were less likely to have had a stroke than nonusers until college, when the two lines converged.
Based on that chart, it looks like marijuana reduces strokes. Except that correlation doesn't prove causation.
The study authors, however, wanted to use these numbers to support the opposite conclusion. First, they replaced the actual data with a logistic regression. That yielded this chart.
But the data still wouldn't confess. Marijuana users still appear healthier. So the authors kept going. They replaced their regression lines with 95 percent confidence intervals, which are used to show the probability that a population parameter will fall between a set of values. Now there seems to be some doubt about whether marijuana is a benefit or a harm, because the blue is dipping below the orange at points.
Using confidence intervals is inappropriate.
Confidence intervals are useful where we're uncertain about data, but we know exactly what the data are in this case. The uncertainty is not in the original numbers; it was introduced by the researchers when they forced logistic regression on the data.
The authors kept going, adding more variables to the logistic regression, including age, sex, race, alcohol use, smoking history, body mass index, physical activity, and diabetes. In all cases, logistic regression obscured the actual data. It didn't add any information. If marijuana users have fewer cardiovascular problems than nonusers in all subgroups, you don't need logistic regression or anything else to refute that marijuana use is the cause of cardiovascular problems.
The logistic regressions still failed to show a statistically significant increased cardiovascular risk for marijuana users. So next the authors excluded inconvenient data.
When they added age as a test of stroke frequency and marijuana, here's what they came up with.
Note that for obvious reasons stroke frequency drops with increased marijuana use: Older people smoke less weed and have more strokes. That doesn't prove marijuana is good for cardiovascular health. But it does indicate that marijuana is not a major cause of cardiovascular problems, which mainly occur in age groups where marijuana use is rare, while most marijuana use happens in age groups where cardiovascular problems are rare.
Those two points in the upper left represent people older than 74, who had a high rate of strokes and low rate of marijuana use. This population helps to undercut the authors' conclusion that marijuana causes cardiovascular problems. So the authors excluded these two data points from the study, even though they're as meaningful as any other data.
How did the authors explain the decision to throw out relevant data? They asserted that these groups, which have the highest rate of cardiovascular problems, didn't use enough marijuana to be suitable for the study.
That's like trying to prove that college calculus causes binge drinking, but excluding frat boys on the grounds that few of them are math majors. It's appalling. But ignoring contrary data did at last get the confession that the researchers wanted.
I could keep going. For example, the authors didn't report the accuracy of their confidence intervals; they excluded all the survey data from 1988 to 2015 and after 2020 without explaining why; they failed to distinguish between edibles and smoking; they didn't pre-register their hypothesis or use a holdout sample, which makes it impossible to evaluate if their findings were statistically significant; and so on.
So what does this all mean? That prestigious journals publish obvious nonsense; peer review doesn't filter out major errors; partisans and journalists cite papers that match their preferred narrative without reading them, understanding them, or caring about their validity; and researchers are rewarded for Potemkin studies, not the pursuit of truth.
The biggest scandal in academia isn't outright fraud, but these conventional methods.
We're haunted by the chemical boogeyman and the moral panic he engenders.
The only way to stop him is with critical analysis. Statistics should be a way to challenge conventional wisdom and combat irrational fear.
The weapons of science have been turned against us.
Photo Credits: imageBROKER/Jochen Eckel/Newscom
Music Credits: Enids Theme by Night Rider 87, Exalted by Night Rider 87, Shadows in Motion by The Magnetic Buzz, Bionic by Gruber, Get Lost in You by Ben Fox
Footage Credits: "Anti-marijuana television ads through the years" by The New Yorker
- Video Editor: Adani Samat
- Motion Graphics: Regan Taylor
- Audio Production: Ian Keyser
The post Does Weed Cause Strokes and Heart Attacks? appeared first on Reason.com.
