Authors: Brynne Boeck, Cara J. Westmark
Categories: Brief Report, bibliometrics, folate, folic acid, food fortification, neural tube defects, rigor and reproducibility
Source: Nutrients
Doi: 10.3390/nu16152503
The health benefits of vitamin B9 (folate) are well documented, particularly in regard to neural tube defects during pregnancy; however, much remains to be learned regarding the health effects and risks of consuming folic acid supplements and foods fortified with folic acid. In 2020, our laboratory conducted a population-based analysis of the Food Fortification Initiative (FFI) dataset to determine the strength of the evidence regarding the prevalence of neural tube defects (NTD) at the national level in response to mandatory fortification of cereal grains with folic acid. We found a very weak correlation between the prevalence of NTDs and the level of folic acid fortification irrespective of the cereal grain fortified (wheat, maize, or rice). We found a strong linear relationship between reduced NTDs and higher socioeconomic status (SES). Our paper incited a debate on the proper statistics to employ for population-level data. Subsequently, there has been a large number of erroneous citations to our original work. The objective here was to conduct a bibliometric analysis to quantitate the accuracy of citations to Murphy and Westmark’s publication entitled, “Folic Acid Fortification and Neural Tube Defect Risk: Analysis of the Food Fortification Initiative Dataset”. We found a 70% inaccuracy rate. These findings highlight the dire need for increased rigor in citing scientific literature, particularly in regard to biomedical research that directly impacts public health policy.
Keywords: bibliometrics, folate, folic acid, food fortification, neural tube defects, rigor and reproducibility
Neural tube defects (NTDs) are serious birth defects that affect the brain and spinal cord and can result in infant mortality or serious disability. The prevalence of NTDs is in the range of 4–32 per 10,000 births with a mean of 13 [1], and 0.3–199.4 per 10,000 births in another study [2]. Adequate folate in the diet during the periconceptual period may reduce the incidence of NTDs by 50% [3]. In 1998, the United States of America (U.S.A.) was the first country to mandate national fortification of cereal grains with folic acid to reduce NTDs. As of November of 2023, 94 countries have mandatory cereal grain fortification legislation [4]. It is estimated that 97% of industrial-scale wheat flour was fortified in the Americas, 31% in Africa, 44% in the Eastern Mediterranean, 21% in Southeast Asia, 6% in Europe, and 4% in Western Pacific regions in 2007 [4,5]. While substantial evidence supports the role of folate in preventing NTDs, the efficacy of fortification of food products with folic acid is debated. Folic acid, which is a synthetic form of folate used for food fortification and vitamin supplements, has a different chemical structure than natural folates found in fruits and vegetables and needs to be converted into 5-methylenetetrahydrofolate, the major circulating form of folate for use by the body as vitamin B9.
In 2020, we published a study in Nutrients comparing NTD prevalence in countries with versus without national folic acid fortification policies using the publicly available Food Fortification Initiative (FFI) dataset [1]. The FFI is a global group that provides technical assistance to governments regarding food fortification to address micronutrient deficiencies. The rationale was that fortification would directly affect the majority of inhabitants in a country. The null hypothesis was that there would be no association between national folic acid fortification and the prevalence of NTDs. The alternative hypothesis was that national folic acid fortification alters the prevalence of NTDs. The primary endpoint of interest was the prevalence of NTDs. The primary predictor variable was folic acid fortification. The average prevalence of NTDs per 10,000 births was calculated as a function of folic acid fortification after binning countries into groups based on which cereal grain was fortified. We found an equivalent average as well as a range of values for NTDs per 10,000 births with and without fortification. We also stratified the FFI NTD data based on national economic indicators and found a strong linear correlation between higher socioeconomic status (SES) and reduced NTDs. Others have also shown correlations between better SES and reduced prevalence of NTDs [2,6]. In the absence of prospective monitoring, it is not possible to determine cause and effect; however, the cumulative data strongly suggest that higher SES, not fortification of cereal grains with folic acid, contributes to reduced prevalence of NTDs.
Food fortification is a major public health issue that has been surrounded by controversy. Kancherla et al. disagreed with the statistical analysis and conclusions of our paper, which are discussed in detail in [7,8]. In our reply [8], we pointed out that a strong case can be made that proponents of national food fortification commit exception fallacies in their analyses. Exception fallacies occur when conclusions about a group of people are based on data from individual cases. In 2022, in the U.S.A., the number of women of childbearing age, 15–44, was 65 million out of a total of 333 million people, which is 20% of the total population. Not all women and their babies are deficient in folate and not all will benefit, and some may actually be harmed, from excess consumption of folic acid dependent on dietary patterns and genetics. Folic acid can interact with medications as well as increase the risk of certain types of cancer including prostate [9,10]. The Cochrane Database of Systematic Reviews assessed the efficacy of folic acid fortification on health outcomes in the overall population as “low certainty” in improving NTD outcomes [11].
The concern addressed in this paper is the high incidence of inaccurate citations regarding our 2020 study. The research process is dependent on the accurate citation of prior scientific findings. Examples of citation inaccuracies include (1) selective citation, or choosing references in an arbitrary fashion; (2) citation bias, which is the preferential citing of positive findings; (3) secondary citation, or not citing the original source; (4) incorrect/opposite conclusion, which occurs when inaccurate or missing information is cited, for example, citing an article presenting the opposite conclusion referred to in the study; and (5) fact not found, which occurs when an cited article does not mention the cited claim [12]. Improper and incomplete referencing of the prior related literature impedes biomedical research and can substantiate poor public health policies. The objective here was to conduct a bibliometric analysis to quantitate the accuracy of citations to Murphy and Westmark’s publication entitled, “Folic Acid Fortification and Neural Tube Defect Risk: Analysis of the Food Fortification Initiative Dataset” [1].
The study design was a content analysis of citations to Murphy and Westmark, 2020 [1]. The search strategy involved retrieval of all citations from Google Scholar at scholar.google.com from the time of publication, 18 January 2020, to 30 May 2024, using the title of the paper. There was a total of 52 citations [6,7,8,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60]. Inclusion criteria included available full texts. One citation was excluded because it was a pre-print and the published text was available (i.e., duplicate publication) [50]. One citation was a book chapter and was excluded because the University of Wisconsin-Madison Library could not retrieve a copy for review [60]. Google Translate was employed to interpret manuscripts in non-English languages.
The citing articles were arranged in alphabetical order by the first author’s last name in Table 1. The extracted data the name of the first author, institution, country, year work was published, name of the journal, title of the paper, and copy of the text citing Murphy & Westmark, 2020 in quotes. Notes were added after the quoted material to clarify discrepancies between the citations and the cited work. Both authors reviewed and graded the 50 included texts for citation accuracy. Inaccurate citations were further reviewed to determine the type of error (selective citation, citation bias, secondary citation, incorrect/opposite finding, and fact not found), where secondary citations were considered less severe and other types of errors were labeled as serious. The entries in Table 1 were color-coded and labeled with the type of error per the figure legend. The data were analyzed in accordance with STROBE guidelines and the BIBLIO checklist [61,62]. A copy of the BIBLIO checklist is provided in Table S1. Percentages were computed to describe the results. To statistically test for differences in accurate citation rates, Fisher Exact tests were used and p values were reported. Statistical significance was defined as p < 0.05.
The paper, “Folic Acid Fortification and Neural Tube Defect Risk: Analysis of the Food Fortification Initiative Dataset” by Murphy and Westmark, averaged 1 citation per month since its publication in January 2020 (52 citations). Here, we found a 70% inaccuracy rate in citing this work (Table 1), which is statistically significantly higher than an inaccuracy of 12.3% for all citations from 19 other papers published by the same laboratory in the same timeframe [130 non-duplicated full-text publications 88% correct; 2.3% mis-citations; 10% secondary citation errors]; p < 0.0001 [8,14,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79]. Others have reported citation inaccuracy at 25% in medical journals including half major and half minor/secondary citation errors [80]. The serious citation errors in Murphy and Westmark, 2020 [1], included 18 papers with selective citation errors, 12 papers with citation bias errors, 21 papers with incorrect/opposite finding errors, and 6 papers with fact not found errors within 25 publications, i.e., half of the papers evaluated contained one or more major errors. Secondary citation errors were found in 15 publications. The errors in 19 other laboratory publications were predominantly secondary citation errors. This bibliometric analysis only includes publications that cited Murphy and Westmark, 2020 [1]. We did not perform a comprehensive analysis of the 294 citations, between 18 January 2020 and 30 May 2024, in PubMed with the search terms “folic acid AND fortification” to determine errors in selective citation where the Murphy and Westmark, 2020 paper should have been cited to discuss alternate findings but was not. Of interest, publications from countries with mandatory legislation regarding the fortification of wheat, as of November 2023 (FFI website [4]), had an increased number of papers with serious citation errors, i.e., 64% of the papers in which the corresponding author was from a country with mandatory fortification had serious errors, whereas 77% of papers from authors in countries without mandatory fortification cited Murphy and Westmark, 2020 correctly, p = 0.038 (note self-citations excluded from analysis).
Seven of the citations were from theses or dissertations, and all contained inaccuracies (five serious errors and two secondary citation errors). Removing the trainee theses and dissertations from the citation pool resulted in 43 papers with a 65% overall error rate and a 47% serious error rate in citations, which is not statistically different from the trainee theses and dissertations with an overall error rate of 100% (p = 0.087) and a serious error rate of 71% (p = 0.42).
These findings highlight the dire need for increased rigor in citing the scientific literature, particularly in regard to biomedical research that directly impacts public health policy. NIH promotes increased rigor and reproducibility in biomedical research through multiple initiatives. Per their website, “Two of the cornerstones of science advancement are rigor in designing and performing scientific research and the ability to reproduce biomedical research findings” [81]. The National Library of Medicine lists Research Reporting Guidelines and Initiatives by Organizations that provide advice from 50 organizations for reporting research methods and findings [82]. Advice is provided on areas including formatting style (AMA Manual of Style and APA Style), animal metrics (ARRIVE), clinical trials (ASSERT, CONSORT), case reports (CARE), systematic reviews and meta-analyses (PRISMA), observational research (STROBE), etc. Per NIH Principles and Guidelines for Reporting Preclinical Research, transparency in reporting should include standards, replicates, statistics, randomization, blinding, sample size estimation, and inclusion and exclusion criteria. A clear and transparent account of what and how research was performed is imperative to reproduce findings and assess the rigor of a study. We assert that a third cornerstone, i.e., accurate citation of the prior literature, is also paramount to rigorous biomedical research, which forms the foundation for public health policy.
Our bibliometric analysis found a 70% inaccuracy rate in citing Murphy and Westmark, 2020, with half of the papers containing serious citation errors. It is perplexing that numerous authors chose to inaccurately cite rather than discuss work that contradicted their findings. It is also disturbing that seven of the citations were from trainee theses or dissertations, including 71% with serious errors and the remaining 29% with secondary citation errors. With the limited number of trainee publications, the error rate approached but was not statistically different from the other publications (p = 0.087) There is a dearth of formal training in scientific writing in graduate school programs. Training may be focused on format and style, not content or accurate reflection of the findings. Students typically learn to write and cite scientific research papers after reading and evaluating the primary literature and through mentor feedback. A 100% error rate is of concern in regard to the quality of literature analysis skills and mentor oversight of writing in graduate training programs. Solid scientific writing requires more than good grammar, appropriate word choice, and telling a coherent story; it requires rigorous analysis and presentation of the prior literature.
Appropriate citation of the prior literature is imperative for research, and, in particular, for science informing public health issues. While our ecological study on population-level efficacy of food fortification with folic acid did not show a benefit [1], and we are unaware of any appropriately controlled population-level study showing efficacy, the biology of folic acid has been extensively studied and there is ample evidence of potential adverse effects [83,84]. Numerous reports indicating a decline in the prevalence of NTDs with mandatory folic acid fortification do not consider that NTDs were declining without fortification [85,86] and do not include a comparison with a non-fortification group during the same time period.
Inaccurate citation, particularly through opposite conclusions, buries science that needs to be debated and impedes the discovery and implementation of efficacious public health policies. For example, public health measures targeting at-risk populations to increase education on adequate nutrition in the pre-conception period as well as access to healthy foods could reduce NTDs. In the Herter-Aeberli study, only 25% of women received information from their gynecologist about folate deficiency and the risk of birth defects before they became pregnant [24]. Only 7% of the women received information about nutrition prior to their pregnancy [24]. Higher SES status exerts a significant effect on folic acid status and NTD risk. We found a strong linear relationship between reduced NTDs and higher socioeconomic status (SES) [1]. Herter-Aeberli and colleagues demonstrated that education was an important predictor of the timing during pregnancy when women took supplements [24]. Specifically, post-secondary education was associated with an increased use of supplements prior to pregnancy, i.e., 41% versus 16% of women without a post-secondary degree [24]. Likewise, higher income was associated with recommended supplement use, i.e., 43% versus 29% of women in the lower income class [24]. Others show that neighborhood deprivation, i.e., lower SES, is associated with increased NTDs [6,87].
A personalized medicine approach in regard to folate and folic acid treatment is needed. There are several vulnerable populations that may be adversely affected by over-exposure to folic acid, such as the elderly who have low vitamin B12 levels, those taking medications that interact with folic acid, and those with certain methylenetetrahydrofolate reductase (MTHFR) polymorphisms. There is potential for nervous system damage in persons with low or insufficient B12 who are exposed to a high intake of folic acid [88,89]. For example, low vitamin B12 in the elderly in conjunction with higher serum folate is associated with anemia and cognitive impairment [90,91,92,93]. Folic acid interacts with 35 medications including chemotherapy, seizure, and cholesterol drugs [10]. Folic acid also interacts with genetic mutations, for example, MTHFR, which is a critical enzyme in folate metabolism affecting DNA methylation, synthesis, and repair.
It is worth discussing a potential mechanism through which folic acid could affect a substantial portion of the population. The cytosine-to-thymine substitution at base pair 677 (C677T) of the MTHFR gene results in the substitution of a valine codon for the evolutionarily conserved alanine residue. This genetic variant is common in North America, where approximately 12% of individuals are homozygous for the C677T allele and 40–45% are heterozygous [94,95]. Persons with the C677T polymorphism exhibit decreased enzymatic activity of MTHFR, which is the final enzyme in a multi-step pathway that converts folic acid to the active folate metabolite 5-methyltetrahydrofolate (5-MTHF), the form that can be transported into cells and across the blood–brain barrier. With the decreased activity of MTHFR(C577T), the final conversion step of folic acid to 5-MTHF may not be complete, leading to high levels of unmetabolized folic acid (UMFA) and increased homocysteine levels.
UMFA was detected in 38% of older adults in the National Health and Nutrition Examination Survey (NHANES) U.S.A. study population after fasting [96]. A double-blind, randomized controlled trial in Australia found more UMFA in serum from women consuming an 800 μg/day folic acid supplement from 12–16 weeks gestation until 36 weeks of gestation compared with the control group [50]. Moderately high folic acid intake has negative effects on embryonic development in mice with an increased prevalence of embryo loss and defects in heart development [97]. In adult rats, folic acid promotes spinal axon regeneration albeit in a bell-shaped dose-dependent and methylation-dependent manner with effects that may persist over several generations [98]. Possible repercussions of excessive folic acid consumption are increased UMFA levels, competition with natural folate for binding to folate receptors, downregulation of intestinal and renal folate uptake, and altered epigenetic programming, which has been postulated to contribute to autism prevalence, NTDs, and sperm degradation. [99,100,101,102,103]. In postmenopausal women with higher RBC folate, altered epigenetic programming in the form of lower DNA methylation was observed in the post-folic acid fortification period [104]. We currently lack clinical data regarding UMFA and neuronal tube defects. Using the PubMed search terms “unmetabolized folic acid” AND “clinical trial” returns 21 publications, but none assess NTDs.
The rates of autism and Alzheimer’s disease have increased substantially since the late 1990s when folic acid fortification was mandated in the U.S.A. [105]. It has been hypothesized that enhanced maternal/fetal folate status during pregnancy alters natural selection for autism [106]. Folate stimulates the expression of secretases involved in the non-amyloidogenic processing of amyloid protein precursor (APP) (ADAM9 and ADAM10) while inhibiting the expression of the amyloidogenic secretase BACE1 [107]. Furthermore, hyperhomocysteinemia increases amyloid-beta levels [108]. These data suggest that excess folic acid could interfere with APP processing pathways, which are implicated in Alzheimer’s disease, fragile X syndrome (FXS), autism, and other neurological disorders [109,110,111]. Meta-analyses indicate a significant association between the MTHFR C677T polymorphism and the risk of NTD pregnancy, autism, and Alzheimer’s disease [112,113,114]. Folic acid deficiency is associated with diverse neuropsychiatric symptoms, yet treatment with folic acid increases psychotic behavior in psychiatric patients [115]. Thus, there is a need to determine optimal folate dosing for the prevention of NTDs while avoiding other neurological disorders and cancer.
It is past time to reconsider public health policy regarding fortification of food with folic acid. There is a lack of prospective monitoring of adverse health effects associated with the consumption of folic acid and concerns surrounding exceeding the upper tolerable limit. Mandatory folic acid fortification in the U.S.A. increased average intake by twice what was projected [116]. Despite mandatory fortification of cereal grains with folic acid in the U.S.A., the estimated prevalence of folate insufficiency based on red blood cell (RBC) folate concentration in women of reproductive age was 23% for NHANES 2007–2012 participants [117,118]. Thus, a large portion of the target population is deficient in folic acid despite fortification and a large portion is at risk for adverse health events in response to fortification.
The Code of Federal Regulations (CFR)—Title 21—Food and Drugs §101.79 states the following regarding health claims associated with folate and NTDs, “The available data show that diets adequate in folate may reduce the risk of neural tube defects. The strongest evidence for this relationship comes from an intervention study by the Medical Research Council of the United Kingdom that showed that women at risk of recurrence of a neural tube defect pregnancy who consumed a supplement containing 4 milligrams (mg)(4000 micrograms (mcg)) folic acid daily before conception and continuing into early pregnancy had a reduced risk of having a child with a neural tube defect. (Products containing this level of folic acid are drugs). In addition, based on its review of a Hungarian intervention trial that reported periconceptional use of a multivitamin and multimineral preparation containing 800 mcg (0.8 mg) of folic acid, and its review of the observational studies that reported periconceptional use of multivitamins containing 0 to 1000 mcg of folic acid, the Food and Drug Administration concluded that most of these studies had results consistent with the conclusion that folate, at levels attainable in usual diets, may reduce the risk of neural tube defects”. Please note that these regulations from the FDA: (1) cite the benefits of supplements not fortification; (2) regard folic acid supplementation studied in at-risk populations, i.e., women at risk for a second NTD-pregnancy; (3) state that products containing high levels of folic acid are considered drugs; dose is not known with fortification as individuals are exposed to varied quantities and types of food; (4) state that supplementation occurred daily before and during early pregnancy, not lifelong; and (5) infer that folate levels adequate to reduce NTDs can be obtained from a typical diet.
Recommendations should also consider potential interactions with agrochemicals. National fortification of cereal grains with folic acid commenced in 1998 at the same time genetically modified foods were introduced into the food supply in the U.S.A. There is a strong association between the increased use of glyphosate (the active ingredient in RoundUp Ready^®^) on genetically modified soy and corn crops and the prevalence of numerous medical conditions including autism [105]. The gut bacteria that metabolize folate are strains of Lactobacilli and Bifidobacteria [119], which are microbes that are preferentially killed by glyphosate [120,121].
Regarding study limitations, a confounding issue of our original analysis demonstrating a reduced prevalence of NTDs with higher SES is that there could be increased maternal use of folic acid supplements during pre-conception and pregnancy with higher education level and increased health knoweldge. The strength of this study is the transparent, quantitative approach to assessing citation accuracy.
In conclusion, accurate citation of scientific findings needs to be included with robust and unbiased experimental design, methodology, analysis, interpretation, and reporting of results as rigor variables in the conduct of science. Particularly troubling are opposite conclusion errors in citation where the opposing data should be discussed instead of misrepresented. The lack of rigor in accurately citing the scientific literature impedes the promotion and implementation of effective public health policies. An honest and critical evaluation of the folic acid fortification literature and admittance of shortcomings in efficacy evaluation is long overdue. National fortification policies were instituted to prevent NTDs during pregnancy; however, the potential effects on miscarriages, autism, Alzheimer’s disease, and cancer would warrant a “first, do no harm” approach. In the absence of a monitoring program at the individual subject level, it is impossible to discern the full benefit or consequences of national folic acid fortification policies. Many individuals are at risk for exceeding the safe upper limit, and many are at risk for deficiency because of food choices and availability. A failure to evaluate the health effects of fortification prospectively will negatively impact vulnerable populations. Overall, we have not witnessed the projected 50% reduction in NTDs in the U.S.A. in response to the national fortification of cereal grains with folic acid, and current fortification levels present health concerns for large sub-groups of the general population. A one-size-fits-all strategy to medicine, and food is medicine, is not working. There is an urgent need for a personalized medicine approach with regard to folic acid.
The authors thank Pamela Westmark for the critical review of this manuscript.
The following supporting information can be downloaded https://www.mdpi.com/article/10.3390/nu16152503/s1. Table S1: BIBLIO checklist for reporting bibliometric reviews of the biomedical literature.
Conceptualization, C.J.W.; methodology, C.J.W.; formal analysis, B.B. and C.J.W.; resources, C.J.W.; data curation, B.B. and C.J.W.; writing—original draft preparation, C.J.W.; writing—review and editing, B.B. and C.J.W.; visualization, C.J.W.; supervision, C.J.W.; project administration, C.J.W.; funding acquisition, C.J.W. All authors have read and agreed to the published version of the manuscript.
Not applicable.
Not applicable.
The original contributions presented in this study are included in this article; further inquiries can be directed to the corresponding author.
The authors declare no conflicts of interest. The funders had no role in the design of this study; in the collection, analyses, or interpretation of data; in the writing of this manuscript; or in the decision to publish the results.
This research was funded by the USDA, grant number 2018-67001-28266.
The original contributions presented in this study are included in this article; further inquiries can be directed to the corresponding author.