Making Sense of Nutrition Research

How Accurate Are Questionnaires?

University of Leeds psychologist John Blundell writes in an issue of the American Journal of Clinical Nutrition "it is now freely admitted that most databases do not inform us well about the actual foods that individuals habitually consume because of the massive problem of misreporting."

David Allison, an associate research scientist at St. Luke's/Roosevelt Hospital Center in New York as well as an associate professor of medical psychology at Columbia University's College of Physicians and Surgeons, puts it more bluntly, "The measurements by questionnaire of what people eat are just terrible."

To drive home his point, he notes that, "I actually know two men who are in the Nurses' Health Study. These guys confessed to me that their wives are nurses, and they [the husbands] fill out the food questionnaires. 'Hey, Honey, how often do we have pasta?'" Allison says one of his acquaintances will call out to his wife in another room as he completes the food frequency questionnaire that's sent to their home every two years.

Walter Willett, chair of the Department of Nutrition at the Harvard School of Public Health, would be the first to admit that when you conduct research on 120,000 women, which is roughly the number of participants in the Nurses' Health Study, it's impossible to keep tabs on every single one. He points out, however, that even with their limitations, studies that revolve around food frequency questionnaires can take steps to minimize error. The Harvard study, he says, has taken more steps than any other.

For instance, researchers there have measured blood levels of nutrients like ]]>folic acid]]> and vitamin E in small subsets of subjects to get an idea of whether the foods that people say they eat are indeed the foods they're eating.

"There's not 100% match," Willett concedes. You can't tell exactly how much people are consuming various foods. But, he says, those with, say, the highest blood levels of folic acid also tend to be the ones who report eating the highest amounts of folic acid-containing foods. Those with the lowest levels tend to be the ones who report eating the least amount of folic acid-rich foods. In other words, you can certainly spot trends in people's food intake that could be scientifically important for seeing links between people's diets and their health status.

The Nurses' Health Study has also employed dietitians to teach small groups of women how to weigh and measure their foods to ascertain the level of match-up between reported consumption and actual consumption. And, remarks Willett, because the study has been going on for 20 years and asks women to report their eating habits every two years, it "accounts for changes over time," whereas most studies involving food frequency questionnaires cannot do that. It's a "business of imperfect measurements," he comments, but he says that the extra steps the Harvard researchers make do help to cut down on error.

People Think They Eat Less Than They Do

That's Harvard, however. "Protective" measures are not always taken. And either way, it can get quite messy out there. Blundell, for his part, cites a study in which a group of obese men underreported their typical calorie intake by 36%.

Truth be told, underreporting of calories is an old story; nutrition researchers have known for years that overweight people "slash" calories from their daily diets when asked in studies how much they eat. But now they're underreporting fat consumption too—as the obese men did.

Blundell isn't surprised. "The more we tell people that they should reduce the amount of fat in their diets," he writes, "the more they tell us how little of it they eat," even though actual intakes do not appear to change.

Referring to all the room for error in study subjects' acknowledgment of what and how much they eat, Blundell says, "we should be cautious about making too many doctrinaire statements on nutrition-disease relations on the basis of data that may be seriously flawed."

It's an Association—Not Cause and Effect

And it gets more complicated still.

People who are asked to report what they eat in studies are followed to see what diseases they develop. These observational studies, can, at best, report only associations between foods and illnesses. By their very nature, they could never tease out cause and effect, even if people could accurately report their food intake.

Take, for example, a study reported in The Washington Post, The New York Times, and a number of other large newspapers some years ago. The Washington Post headline, similar to the others, read, "Study Links Hot Dogs, Cancer: Ingestion by Children Boosts Leukemia Risk, Report Says." The first sentence went as follows: "Children who eat more than 12 hot dogs per month have nine times the normal risk of developing childhood leukemia."

It sounds as though hot dogs cause leukemia when all that was found was that hot dogs and leukemia somehow went together. Maybe it was the buns—or the mustard or relish—or something entirely unrelated to the children's eating habits. Add to that the fact that the study relied entirely on parents' recollections of how many hot dogs their children had eaten (recollections that sometimes went back years), and the link becomes more tenuous still.

Okay, maybe that story didn't hit too close to home. Chances are you're not feeding your kids 12 hot dogs a month, anyway. But what about something like vitamin E supplements? Several large-scale studies that have asked people what they typically eat have found that those who get a lot of vitamin E—via supplements—are less likely than others to die of heart disease. Based on the results, sales of vitamin E supplements soared.

Subsequently, a series of much more meaningful studies were performed on vitamin E to discover whether it really prevents heart disease. These were intervention trials, in which people were given either vitamin E or a placebo. Studies of this type can determine actual cause and effect. The upshot: vitamin E does not have any effect on heart disease. Apparently, the people who happened to take more vitamin E in the observational studies were healthier for other reasons–maybe they also exercised more, for instance.

The Role of the Media

For better or for worse, however, the What-Do-You-Eat research efforts often get more play in the media. Alice Lichtenstein, a heart disease researcher at Tufts University, thinks those types of studies get "a lot more coverage because they're simple to understand, in a sense. They ask relevant questions that the everyday person is interested in," such as what type of eaters get cancer and what type get heart disease? "But the everyday person," she says, "isn't always thinking of the difference between a factor 'causing' a disease versus simply having an 'association' with the disease."

Sometimes scientists aren't thinking of it, either. James Fleet, the director of the Graduate Program in Nutrition at the University of North Carolina at Greensboro, says that scientists can get carried away by their own results and read more into them than is there.

A newspaper reader or television viewer, he comments, "should check to see that the researcher who conducts studies looking for associations between dietary patterns and diseases is very cautious about relaying a new idea, saying how this or that might help if there are more data in the future." If the researcher is too gung ho in his or her quotes, Fleet says, that's a red flag.

Not that Fleet or any of the other researchers believe food questionnaire research isn't important.

"It's highly important," Fleet says, because it "generates new hypotheses to test in more controlled settings," for instance, the hypothesis that large doses of vitamin E may help prevent heart disease. "It just shouldn't be represented as the answer," he explains.

How Should You Interpret It All?

It should, in fact, be taken "with a considerable grain of salt," by the lay community, says Allison. While findings from such research are often interpreted as "you should do something about this now," he says, "that's not where it's at. It doesn't produce guidance about what to do today —eat more of this, less of that."

Tuft's Lichtenstein agrees. What people should think when they read about such research, she says, is, "Well, gee, that's interesting. I'll file that in the back of my head—wait for more studies to come out."

If more studies do come out and the data start to crystallize, says Allison, "the USDA will make a statement. The American Heart Association will make a statement." In other words, health-promoting organizations will provide dietary guidance based on the totality of evidence rather than on the results of this or that study in which people couldn't accurately report what they ate, anyway.

Harvard's Willett agrees that no one study ever tells the whole story. But he thinks research such as the Nurses' Health Study can carry more weight than it's sometimes given credit for in that it can help to test hypotheses as well as generate them.

For instance, he says, it has been known for a long time that women in Japan are less likely to get breast cancer than women in the US. And it was thought that maybe the low fat intake of the Japanese women is what is contributing to their reduced risk. But the lifestyles and genetic backgrounds of women in the two countries are different enough that any number of factors could be influencing breast cancer risk.

The Nurses' Health Study, in contrast, eliminates a lot of the "static" created by those differences by looking at tens of thousands of women whose lifestyles and genetic backgrounds are more similar. And what it has found is that how much or how little fat American women eat does not appear to affect their chances of getting breast cancer; it doesn't seem to be a low-fat diet that's protecting Japanese women from that disease.

Granted, the best way to test the fat/breast cancer theory would be to put some women on a high-fat diet and a similar group on a low-fat diet and follow them for many years to see what happens. Research that approximates that approach is, in fact, going on now. But, Willett says, the trial will take 15 years altogether and cost a billion dollars. In other words, he remarks, that type of investigation is just not always going to be feasible, so "probably, in many instances, there will be no better information available" than that gained from investigations such as the Nurses' Health Study, which can be conducted relatively inexpensively.

Still, you have to be cautious about interpreting the results of food frequency questionnaire research—including the best of that research—as a call to dietary action. Even nutritional epidemiologist Katherine Tucker of Tufts, who conducts studies by asking people what they eat, sees it that way. "A new finding is news," she says, "and I think it should get out there. But it doesn't mean you should take it as dietary advice. Look at the body of accumulated evidence."