PLoS Medicine | www.plosmedicine.org 1129
Correspondence
June 2007 | Volume 4 | Issue 6 | e211 | e225
Reporting of Systematic Reviews:
The Challenge of Genetic Association
Studies
Muin J. Khoury, Julian Little, Julian Higgins, John P. A.
Ioannidis, Marta Gwinn
We applaud PLoS editors for their commitment to publishing
high-quality systematic reviews (SRs) [1]. Moher et al. [2]
clearly documented the inconsistent quality of reporting
of SRs. With more than 2,500 SRs published every year,
low-quality or outdated reviews may mislead researchers,
providers, and policy makers. The situation could be
improved if more evidence-based reporting guidelines were
agreed upon, developed, and adhered to.
The growing fi eld of genetic associations (GAs) illustrates
the urgent need for transparent SRs and meta-analyses.
Already, thousands of articles on GAs have been published,
and the application of high-throughput genotyping methods
may exponentially increase the number of reported
associations [3]. Selective reporting of large numbers of false-
positive associations could undermine the fi eld and interfere
with our ability to translate advances in genomics into clinical
practice.
To address these problems, the Human Genome
Epidemiology Network (HuGENet) was started as a global
collaboration to strengthen methods of analysis and reporting
of GAs and to develop a reliable knowledge base on the
association between genetic variation and human diseases
[4]. Between 2001 and 2006, the HuGENet online database
assembled more than 25,000 published articles on GAs and
more than 500 systematic reviews of GAs. Nevertheless, there
are large inconsistencies in the quality of genetic association
studies [5] and in the reporting of SRs of such associations
[6]. In collaboration with several journals, HuGENet
promotes the publication of transparently reported SRs of
gene–disease associations [4]. More than 50 HuGE reviews
have been published over the past six years.
After several HuGENet workshops bringing together
researchers from different fi elds and journal editors, the fi rst
edition of a HuGENet handbook, modeled in part after the
Cochrane handbook of systematic reviews, was published
on the Canadian HuGENet Web site [7]. The handbook
describes methodological issues and outlines steps in
conducting such reviews, including the need for a detailed
protocol. It also discusses meta-analysis methods. We strongly
encourage researchers interested in conducting systematic
reviews of GAs to consult the HuGENet handbook, and adopt
transparent protocols. Retrospective SRs of published data
have limitations, even when properly conducted. Investigators
can advance the fi eld of human genome epidemiology by
conducting prospective meta-analyses and large collaborative
analyses through international consortia. HuGENet has
created a Network of Investigator Networks to help the
growth of such initiatives [8].

Muin J. Khoury (muk1@cdc.gov)
Marta Gwinn
Centers for Disease Control and Prevention
Atlanta, Georgia, United States of America
Julian Little
University of Ottawa
Ottawa, Ontario, Canada
Julian Higgins
Institute of Public Health
Cambridge, United Kingdom
John P. A. Ioannidis
University of Ioannina School of Medicine
Ioannina, Greece
References
1. The PLoS Medicine Editors (2007) Many reviews are systematic but some are
more transparent and completely reported than others. PLoS Med 4: e147.
doi:10.1371/journal.pmed.0040147
2. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007)
Epidemiology and reporting characteristics of systematic reviews. PLoS Med
4: e78. doi: 10.1371/journal.pmed.0040078
3. Khoury MJ, Little J, Gwinn M, Ioannidis JP (2006) On the synthesis and
interpretation of consistent but weak gene-disease associations in the era of
genome-wide association studies. Int J Epidemiol. E-pub 20 December 2006.
4. Centers for Disease Control and Prevention (2007) Human Genome
Epidemiology Network (HuGENet). Available: http://www.cdc.gov/
genomics/hugenet/default.htm. Accessed 24 May 2007.
5. Bogardus ST Jr, Concato J, Feinstein AR (1999) Clinical epidemiological
quality in molecular genetic research: The need for methodological
standards. JAMA 281: 1919–1926.
6. Attia J, Thakkinstian A, D’Este C (2003) Meta analysis of molecular
association studies: Methodologic lessons for genetic epidemiology. J Clin
Epidemiol 56: 297–303.
7. Little J, Higgins J, editors (2006) The HuGENet HuGE review handbook,
version 1.0. Available: http://www.genesens.net/_intranet/doc_nouvelles/
HuGE%20Review%20Handbook%20v11.pdf. Accessed 24 May 2007.
8. Ioannidis JP, Gwinn M, Little J, Higgins JP, Bernstein JL, et al. (2006) A
road map for effi cient and reliable human genome epidemiology. Nat
Genet 38: 3–5.
Citation: Khoury MJ, Little J, Higgins J, Ioannidis JPA, Gwinn M (2007) Reporting of
Systematic Reviews: The challenge of genetic association studies. PLoS Med 4(6):
e211. doi:10.1371/journal.pmed.0040211
Copyright: This is an open-access article distributed under the terms of the
Creative Commons Public Domain Declaration, which stipulates that, once placed
in the public domain, this work may be freely reproduced, distributed, transmitted,
modifi ed, built upon, or otherwise used by anyone for any lawful purpose.
Funding: The authors received no specifi c funding for this article.
Competing Interests: The authors have declared that no competing interests exist.
Reporting of Systematic Reviews:
Better Software Required
Jan Brogger
This is an important paper and editorial [1,2]. Systematic
reviews should be much more widespread, and not only for
randomized clinical trials of clinical treatments. A paper on
an elegant piece of experimental data or on epidemiological
observations would be made all the more interesting if the
fi rst table were a high-quality assessment of previous studies.
In fact, I would suggest that performing a systematic review
should be part of a research protocol for any subject, even
before the study is initiated. However, this paper confi rms my
suspicion that the rising popularity of “systematic reviews” has
not been followed by adherence to methodological rigor.
With this background, I would like to point out one
weakness that may explain part of the current quality defi cit in
some systematic reviews. There is a substantial lack of software
that can assist in an important part of a systematic review:
tracking literature searches and early phase screening. From

PLoS Medicine | www.plosmedicine.org 1130
browsing of the literature and communications with various
Norwegian and Danish Cochrane collaborators (including the
RevMan developers), there seems to be a limited number of
tools for this use. Oftentimes, it is suggested that commercial
reference management software be used, such as the popular
EndNote. These types of software were not designed with
systematic reviews in mind. At later stages of a review,
Cochrane’s RevMan is useful, but not early on.
As far as I have been able to ascertain, there are only
two tools presently available. The fi rst is EPPI-Reviewer
(http:⁄⁄eppi.ioe.ac.uk/cms/Default.aspx?tabid=184), which is
non-profi t, but does not seem to be open source or available
for local deployment. The second is TrialStat’s SRS software
(http://www.trialstat.com), which is commercial and has a
substantial price tag.
I would therefore encourage researchers and institutions
to contribute to the development of open-source tools for
assisting in systematic reviews. I am currently writing such
a simple tool, based on the open-source JabRef package
(http://sourceforge.net/projects/jabref) and would welcome
feedback on perceived needs and other similar projects.

Jan Brogger (jan.brogger@nevro.uib.no)
University of Bergen
Bergen, Norway
References
1. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007)
Epidemiology and reporting characteristics of systematic reviews. PLoS Med
4: e78. doi: 10.1371/journal.pmed.0040078
2. The PLoS Medicine Editors (2007) Many reviews are systematic but some are
more transparent and completely reported than others. PLoS Med 4: e147.
doi:10.1371/journal.pmed.0040147
Citation: Brogger J (2007) Reporting of Systematic Reviews: Better software
required. PLoS Med 4(6): e225. doi:10.1371/journal.pmed.0040225
Copyright: © 2007 Jan Brogger. This is an open-access article distributed under the
terms of the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original author
and source are credited.
Funding: The author received no specifi c funding for this article.
Competing Interests: The author has declared that no competing interests exist.
Neuraminidase Antibodies and H5N1:
Geographic-Dependent Infl uenza
Epidemiology Could Determine Cross-
Protection against Emerging Strains
Jesus F. Bermejo-Martin, David J. Kelvin, Yi Guan,
Honglin Chen, Pilar Perez-Breña, Inmaculada Casas,
Eduardo Arranz, Raul O. de Lejarazu
We have read with great interest the work of Sandbulte et
al. recently published in your journal [1]. In this article, the
authors provide evidence for the existence of cross-immunity
between the neuraminidase of H5N1 viruses and that of
endemic human H1N1 viruses. Age may be an important
determining factor in the development of cross-immunity:
younger people, having a shorter history of H1N1 exposure,
may be disproportionately susceptible to H5N1 infection.
We would like to highlight the infl uence of the geographic-
dependent epidemiological behaviour of infl uenza in the
development of cross-immunity. While Europe, the United
States, and northern Asia experience regular outbreaks
of infl uenza each year, (“seasonal infl uenza”), infl uenza
in tropical regions such as southern China, Vietnam, and
Indonesia tends to be year-round (“non-seasonal” infl uenza).
In consequence, the probability of exposure to infl uenza A in
these regions persists throughout the entire year. Repetitive
contacts with infl uenza wild viruses could promote the
development of cross-immunity against different viral strains.
Even more, it could represent a fortuitous mechanism for
developed natural protection by the close and persistent
exposure of the immune system to infl uenza wild viruses in
regions known for being an important source of emergent
viruses, like southern China.
Results from Sandbulte et al. show that antibodies play a
dominant role in cross-protection. The authors underscore
the possible benefi t of seasonal infl uenza vaccination for
human populations faced with the threat of pandemic H5N1
infl uenza. This idea deserves careful analysis. The main
group at risk for severe complications of seasonal fl u are
people older than 65. In Western countries, this population
is recommended to receive annual vaccinations. Generally
speaking, elder vaccination rates in tropical countries are
far lower than those in Western countries. Even with the low
annual vaccination rate in elders, H5N1 infection is observed
mostly in young people. The existence of sub-clinical or
asymptomatic infections in elderly people cannot be ruled
out, but the reason why there are no described clinical
cases of H5N1 in people older than 40 years is currently
unknown. An age-dependent differential distribution of
avian-type receptors in the upper respiratory tract could
be a possible explanation. On the other hand, Tumpey et
al. [2] demonstrated that mucosal (but not parenteral)
challenges with inactivated or live H3N2 virus protect against
H5N1 infection in mice. These results could have a relevant
consequence: does contact with circulating infl uenza A
via the respiratory tract confer a higher degree of cross-
protection than parenteral exposure to vaccines?
In conclusion, the non-seasonal epidemiological behaviour
of infl uenza in tropical countries could dramatically infl uence
the development of naturally induced cross-immunity against
different infl uenza strains and diminish the risk of severe
disease from new emergent strains in elderly people living
in these countries. The apparent lack of H5N1 cases in the
elderly may be the result of continued exposure to circulating
non-seasonal infl uenza A via mucosal epithelium in the
respiratory tract. Vaccination via the mucosal route could be
a more effi cient way to provide cross-protection against future
pandemic strains than vaccination via the parenteral route. In
this hypothetical scenario, Western countries would be under-
protected.

Jesus F. Bermejo-Martin (bermejo@ped.uva.es)
Eduardo Arranz
Raul O. de Lejarazu
University of Valladolid
Valladolid, Spain
David J. Kelvin
Joint Infl uenza Research Center, Division of Immunology
Shantou University Medical College
Shantou, People’s Republic of China
Division of Experimental Therapeutics
Toronto General Research Institute, University Health Network
Toronto, Ontario, Canada
June 2007 | Volume 4 | Issue 6 | e225 | e212

PLoS Medicine | www.plosmedicine.org 1131
Yi Guan
Honglin Chen
University of Hong Kong
Hong Kong, People’s Republic of China
Pilar Perez-Breña
Inmaculada Casas
National Centre of Microbiology
Majadahonda, Spain
References
1. Sandbulte MR, Jimenez GS, Boon AC, Smith LR, Treanor JJ, et al. (2007)
Cross-reactive neuraminidase antibodies afford partial protection against
H5N1 in mice and are present in unexposed humans. PLoS Med 4: e59.
doi:10.1371/journal.pmed.0040059
2. Tumpey TM, Renshaw M, Clements JD, Katz JM (2001) Mucosal delivery
of inactivated infl uenza vaccine induces B-cell-dependent heterosubtypic
cross-protection against lethal infl uenza A H5N1 virus infection. J Virol 75:
5141–5150.
Citation: Bermejo-Martin JF, Kelvin DJ, Guan Y, Chen H, Perez-Breña P, et al.
(2007) Neuraminidase Antibodies and H5N1: Geographic-dependent infl uenza
epidemiology could determine cross-protection against emerging strains. PLoS
Med 4(6): e212. doi:10.1371/journal.pmed.0040212
Copyright: © 2007 Bermejo-Martin et al. This is an open-access article distributed
under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the
original author and source are credited.
Funding: JFBM is supported by the Fondo de Investigaciones Sanitarias, Ministry of
Health, Spain.
Competing Interests: The authors have declared that no competing interests exist.
Lethal Injection: Let’s Be Honest about
the Death Penalty
Lawrence Bonchek
I don’t favor the death penalty, and I don’t participate
in executions, but I recognize that honorable people can
disagree about the subject, and I don’t consider physicians
who wish to do so—to assure that death comes quickly and
without unusual pain—to be behaving unethically. They
could be seen as serving the interests of both the condemned
and society.
The conundrum about lethal injection persists only
because long-standing American Medical Association
guidelines prohibit physicians from carrying out actual
executions, or even pronouncing death, though they may
certify it—a distinction without a difference. The lack of
physician involvement has resulted in execution protocols
based on outmoded pharmacologic methods, carried out
by inconsistently qualifi ed technicians, with results that are
sometimes ineffective and therefore are understandably
controversial.
Any qualifi ed anesthesiologist could propose more reliable
techniques. It is unreasonable to assert that a condemned
person cannot be executed painlessly, when tens of thousands
of people are anesthetized every single day for surgery with
modern fast-acting anesthetic drugs (propofol, in particular)
that are far more suitable than the outmoded execution drug
thiopental. Induction of surgical anesthesia does occasionally
cause slight injection pain, so how then can it be “cruel and
unusual” to use the same drug and method for the initial step
in executions? Next, potassium cannot fail to stop the heart
instantly and insensibly if given in substantial amounts.
The debate about the death penalty should be conducted
about its morality, not about its methods, because the latter
is merely a surrogate for serious debate. Opponents of the
death penalty (like me) should recognize that it is unwise to
criticize methods alone, because improved methods vitiate
those arguments [1].

Lawrence Bonchek (lbonchek@yahoo.com)
Lancaster General Hospital
Lancaster, Pennsylvania, United States of America
References
1. The PLoS Medicine Editors (2007) Lethal injection is not humane. PLoS
Med 4: e171. doi:10.1371/journal.pmed.0040171
Citation: Bonchek L (2007) Lethal Injection: Let’s be honest about the death
penalty. PLoS Med 4(6): e213. doi:10.1371/journal.pmed.0040213
Copyright: © 2007 Lawrence Bonchek. This is an open-access article distributed
under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the
original author and source are credited.
Funding: The author received no specifi c funding for this article.
Competing Interests: The author has declared that no competing interests exist.
Lethal Injection: Other Views
The PLoS Medicine Editors
The recently published research article on lethal injection [1]
and our editorial commentary [2] both produced a number
of short letters via our electronic reader response system.
There were fi ve letters commenting on the research
article. All were written by strong supporters of the death
penalty who took the view that, as the victims of those who
had been convicted of murder had suffered, the perpetrators
should themselves experience pain as well as execution. We
posted two of these letters on our Web site, but felt the other
three were written in terms that made them unsuitable for
inclusion.
Our editorial attracted 11 letters. Three supported
the views we had expressed, one commented on a small
factual error, and seven were hostile, again focusing on the
desirability of making murderers suffer. We felt that one of
the supportive letters (above) was of particular interest and
we have therefore chosen to highlight it by means of formal
publication in the Correspondence section of the journal. Of
the seven hostile letters, we considered that four were suitable
for posting on the Web site.
We do not intend, within our Correspondence section,
to publish further letters commenting on the research
article or the editorial. However, as with all the articles we
publish, reader responses for the Web site may be submitted
at any time. After a very brief screening for suitability,
reader responses appear on the site within a day or two of
submission.

The PLoS Medicine Editors
The Public Library of Science
San Francisco, California, United States of America
References
1. Zimmers TA, Sheldon J, Lubarsky DA, López-Muñoz F, Waterman L, et al.
(2007) Lethal injection for execution: Chemical asphyxiation? PLoS Med 4:
e156. doi:10.1371/journal.pmed.0040156
2. The PLoS Medicine Editors (2007) Lethal injection is not humane. PLoS
Med 4: e171. doi:10.1371/journal.pmed.0040171
Citation: The PLoS Medicine Editors (2007) Lethal Injection: Other views. PLoS Med
4(6): e224. doi:10.1371/journal.pmed.0040224
June 2007 | Volume 4 | Issue 6 | e212 | e213 | e224

PLoS Medicine | www.plosmedicine.org 1132
Copyright: © 2007 The PLoS Medicine Editors. This is an open-access article
distributed under the terms of the Creative Commons Attribution License, which
permits unrestricted use, distribution, and reproduction in any medium, provided
the original author and source are credited.
Funding: The authors received no specifi c funding for this article.
Competing Interests: The authors have declared that no competing interests exist.
Consent for Genomic Epidemiology in
Developing Countries: Added Human
Subject Protection Also Needed
Robert Reinhard
The authors deserve thanks for laying out decent principles
of communication [1]. But serviceable consent language is
insuffi cient to address all issues of protection. That was the
point of recent workshops held by the National Institutes of
Health to develop a genome-wide association studies program
[2].
Risks associated with personal identifi cation may be
incurred if information is subject to code breaking. Legal
means are available to compel identifi cation, including
across national boundaries. Privacy protections under the
Health Insurance Portability and Accountability Act (HIPAA)
are subject to exceptions, including for law enforcement,
downstream data users, or for other reasons, and are not
available internationally. Even with authorization, the
complexities associated with a repository may frustrate
attempts to achieve meaningful comprehension. Use of
data for purposes other than pharmaceutical product
development or biomedical interventions would be an abuse
resulting perhaps in travel restrictions or discrimination.
For these reasons, safeguards should be added, including:
• Amendments to prevent non-medical health access to
personal identifi cation information;
• Restrictions on recruitment of populations especially
vulnerable to disclosure risks, such as prisoners or
immigrants;
• Prohibitions on disclosure to or use by employers or
third-party payors to deny medical coverage, assign
differential premium risks, restrict access to therapies, or
unfairly discriminate in employment.
Another risk from creation of a genomics repository is
the potential for unjust stigmatization (see for example
[3]). A workable program would state that the data are
appropriate only for limited public health purposes involving
product development or professionally derived biomedical
intervention, and are insupportable for other use or by
political or non-medical entities.
A researcher publishing results based on the genomic data
should state affi rmatively a boilerplate recognition of the
abuse potential for stigmatization. This mechanism could
prevent others from the wayward misappropriation of data
for purposes other than those intended by professionals. The
boilerplate could read:
“Conclusions derived from the genotypic or phenotypic
characterization of individuals, groups, or families in this
[publication] are meaningful or supportable only for the
purpose of biomedical intervention or treatment and
are unethical, insupportable, or inappropriate for use in
other purposes. Use of the data to support any result of
stigmatization, discrimination, or adverse social harm would
constitute a misuse or abuse of the data.”
To increase the connection of benefi ts to participants,
individuals should be given personal opportunities to receive
news reports if they wish and learn of particular clinical trials
directed at their characteristics. If the data are to be used in
the development of pharmaceutical products, users should
also be directed to plan and explain early on how targeted
populations may have reasonable access to treatment or
therapy if the product is successfully brought to market.
These suggestions are consistent with the program outlined
by Senator Barack Obama in the Genomics and Personalized
Medicine Act of 2006 and Senator Olympia Snowe in the
Genetic Information Nondiscrimination Act of 2007 [4,5].
Improved consent: Yes, but linked to and inseparable from
strong protections and added benefi ts for participants.

Robert Reinhard (rreinhard@mofo.com)
San Francisco Vaccine Trials Unit
San Francisco, California, United States of America
References
1. Chokshi DA, Thera MA, Parker M, Diakite M, Makani J, et al. (2007) Valid
consent for genomic epidemiology in developing countries. PLoS Med 4:
e95. doi:10.1371/journal.pmed.0040095
2. National Institutes of Health (2006) Request for information (RFI):
Proposed policy for sharing of data obtained in NIH supported or
conducted genome-wide association studies (GWAS). Available:
http:⁄⁄grants.nih.gov/grants/guide/notice-fi les/NOT-OD-06-094.html.
Accessed 24 May 2007.
3. Council for International Organizations of Medical Sciences (2006) Special
ethical considerations for epidemiological research.
4. US Congress (2006) The Genomics and Personalized Medicine Act of 2006.
S. 3822, 109th Congress, 2d session. Available: http:⁄⁄www.govtrack.us/
congress/billtext.xpd?bill=s109-3822. Accessed 24 May 2007.
5. US Congress (2007) Genetic Information Nondiscrimination Act of 2007.
S. 358, 110th Congress, 1st Session. Available: http:⁄⁄www.govtrack.us/
congress/bill.xpd?bill=s110-358. Accessed 24 May 2007.
Citation: Reinhard R (2007) Consent for Genomic Epidemiology in Developing
Countries: Added human subject protection also needed. PLoS Med 4(6): e214.
doi:10.1371/journal.pmed.0040214
Copyright: © 2007 Robert Reinhard. This is an open-access article distributed under
the terms of the Creative Commons Attribution License, which permits unrestricted
use, distribution, and reproduction in any medium, provided the original author
and source are credited.
Funding: The author received no specifi c funding for this article.
Competing Interests: The author has declared that no competing interests exist.
Why Most Published Research Findings
Are False: Author’s Reply to Goodman
and Greenland
John P. A. Ioannidis
I thank Goodman and Greenland for their interesting
comments [1] on my article [2]. Our methods and results
are practically identical. However, some of my arguments are
misrepresented:
1. I did not “claim that no study or combination of studies
can ever provide convincing evidence.” In the illustrative
examples (Table 4), there is a wide credibility gradient (0.1%
to 85%) for different research designs and settings.
2. I did not assume that all signifi cant p-values are around
0.05. Tables 1–3 and the respective positive predictive value
(PPV) equations can use any p-value (alpha). Nevertheless,
the p = 0.05 threshold is unfortunately entrenched in many
scientifi c fi elds. Almost half of the “positive” fi ndings in
June 2007 | Volume 4 | Issue 6 | e224 | e214 | e215

PLoS Medicine | www.plosmedicine.org 1133
recent observational studies have p-values of 0.01–0.05 [3,4];
most “positive” trials and meta-analyses also have modest p-
values.
3. I provided equations for calculating the credibility of
research fi ndings with or without bias. Even without any bias,
PPV probably remains below 0.50 for most non-randomized,
non-large-scale circumstances. Large trials and meta-analyses
represent a minority of the literature.
4. Figure 1 shows that bias can indeed make a difference.
The proposed modeling has an additional useful feature: As
type I and II errors decrease, PPV(max) = 1 − [u/(R + u)],
meaning that to allow a research fi nding to become more
than 50% credible, we must fi rst reduce bias at least below
the pre-study odds of truth (u less than R). Numerous studies
demonstrate the strong presence of bias across research
designs: indicative reference lists appear in [5–7]. We should
understand bias and minimize it, not ignore it.
5. “Hot fi elds”: Table 3 and Figure 2 present “the
probability that at least one study, among several done on
the same question, claims a statistically signifi cant research
fi nding.” They are not erroneous. Fields with many furtive
competing teams may espouse signifi cance-chasing behaviors,
selectively highlighting “positive” results. Conversely, having
many teams with transparent availability of all results and
integration of data across teams leads to genuine progress.
We need replication, not just discovery [5].
6. The claim by two leading Bayesian methodologists that
a Bayesian approach is somewhat circular and questionable
contradicts Greenland’s own writings: “One misconception
(of many) about Bayesian analyses is that prior distributions
introduce assumptions that are more questionable than
assumptions made by frequentist methods” [8].
7. Empirical data on the refutation rates for various
research designs agree with the estimates obtained in the
proposed modeling [9], not with estimates ignoring bias.
Additional empirical research on these fronts would be very
useful.
Scientifi c investigation is the noblest pursuit. I think we can
improve the respect of the public for researchers by showing
how diffi cult success is. Confi dence in the research enterprise
is probably undermined primarily when we claim that
discoveries are more certain than they really are, and then the
public, scientists, and patients suffer the painful refutations.

John P. A. Ioannidis (jioannid@cc.uoi.gr)
University of Ioannina School of Medicine
Ioannina, Epirus, Greece
References
1. Goodman S, Greenland S (2007) Why most published research fi ndings
are false: Problems in the analysis. PLoS Med 4: e168. doi:10.1371/journal.
pmed.0040168
2. Ioannidis JPA (2005) Why most published research fi ndings are false. PLoS
Med 2: e124. doi:10.1371/journal.pmed.0020124
3. Pocock SJ, Collier TJ, Dandreo KJ, de Stavola BL, Goldman MB, et al.
(2004) Issues in the reporting of epidemiological studies: A survey of recent
practice. BMJ 329: 883.
4. Kavvoura FK, Liberopoulos G, Ioannidis JP (2007) Selection in reported
epidemiological risks: An empirical assessment. PLoS Med 4: e79.
doi:10.1371/journal.pmed.0040079
5. Ioannidis JP (2006) Evolution and translation of research fi ndings: From
bench to where? PLoS Clin Trials 1: e36. doi:10.1371/journal.pctr.0010036
6. Gluud LL (2006) Bias in clinical intervention research. Am J Epidemiol 163:
493–501.
7. The Cochrane Collaboration (2007) Cochrane methodology register.
Available: http://www3.cochrane.org/access_data/cmr/accessDB_cmr.asp.
Accessed 23 May 2007.
8. Greenland S (2006) Bayesian perspectives for epidemiological research: I.
Foundations and basic methods. Int J Epidemiol 35: 765–775.
9. Ioannidis JP (2005) Contradicted and initially stronger effects in highly
cited clinical research. JAMA 294: 218–228.
Citation: Ioannidis JPA (2007) Why Most Published Research Findings Are False:
Author’s reply to Goodman and Greenland. PLoS Med 4(6): e215. doi:10.1371/
journal.pmed.0040215
Copyright: © 2007 John P. A. Ioannidis. This is an open-access article distributed
under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the
original author and source are credited.
Funding: The author received no specifi c funding for this article.
Competing Interests: The author has declared that no competing interests exist.
Biomedical Journals and Global
Poverty: Is HINARI a Step Backwards?
Javier Villafuerte-Gálvez, Walter H. Curioso,
Oscar Gayoso
Much has been written about how open access to biomedical
journals is vital for researchers in developing countries [1],
but so much more needs to be done.
Our experience in Peru with the Health InterNetwork
Access to Research Initiative (HINARI), an initiative managed
by the World Health Organization that helps promote access
to scientifi c information by providing free (or low cost)
online access to major science journals, is not as accessible
as hoped for and, in fact, is getting worse. When HINARI
launched in 2003, it provided access to more than 2,300
major journals in biomedical and related social sciences [2].
In April 2007, we conducted a review of the fi rst 150
science journals available through HINARI with the
highest impact factors on the Science Citation Index [3].
We excluded open-access journals and journals that make
online access free to low-income countries (e.g., The New
England Journal of Medicine, British Medical Journal Publishing
Group). We could not access any of the top fi ve journals from
major publishers such as Nature and Elsevier-Science Direct.
In other words, from the Nature Publishing Group we had
no access to Nature Reviews Cancer, Nature Reviews Immunology,
Nature Reviews Molecular Cell Biology, Nature, or Nature
Medicine, and from Elsevier ScienceDirect we had no access
to Cell, Cancer Cell, Current Opinion in Cell Biology, Immunity,
or Molecular Cell. In addition, we could not access any of the
fi rst-level journals from Blackwell, Oxford Press University,
Lippincott Williams and Wilkins, or Wiley and Sons. In 2003,
all these journals were available.
Our fi ndings support comments received from users over
the last 8–10 months at the main library at Universidad
Peruana Cayetano Heredia (Oscar Gayoso, personal
communication). Students and faculty could not get access
to biomedical journals from Nature, Elsevier-Science Direct,
Blackwell, Oxford Press University, Springer Science,
Lippincott Williams and Wilkins, or Wiley and Sons through
HINARI. The collections of journals from the above-
mentioned publishers together represent approximately
57% (2,118 of 3,741) of journals that were supposed to
be accessible through HINARI, while the remaining 43%
accessible were largely composed of open-access journals or
journals that make online access free to low-income countries.
Moreover, we have found a signifi cant decrease in the
number of users accessing HINARI at our institution. For
example, the number of HINARI users has decreased from
June 2007 | Volume 4 | Issue 6 | e215 | e220

PLoS Medicine | www.plosmedicine.org 1134
12,144 in April 2005 to 5,655 in April 2007, which may refl ect
the loss of impact of the HINARI initiative at our institution.
In contrast, the number of users accessing other databases
such as ProQuest and EBSCO has increased over the last few
months.
Our fi ndings suggest that we not only have access to a
reduced number of biomedical journals on HINARI, but we
also have no access to the biomedical journals that have the
highest impact factors. The HINARI Web site states that it is
still incorporating new journal collections. However, we are
afraid that any addition that will not provide access to major
publishers (such as the Nature Publishing Group, Elsevier
ScienceDirect, or Lippincott Williams and Wilkins) could lack
real impact according to HINARI’s goals.
Since 2003, Peruvian medical students and health
professionals have substantially benefi ted from access to
high-quality scientifi c information through HINARI. Few
medical students and very few researchers in the developing
world can pay the usual fee of US$20–US$45 to download
one article. Not even some private universities in Peru
can afford the minimum journal subscription rates, even
though these subscriptions would help the universities to
become less isolated from global medical research. Having
to pay US$1,000 per year to HINARI has left many public
universities in the provinces of Peru without access because
they cannot afford it. Even for the Peruvian institutions that
are currently paying US$1,000 per year to HINARI, what is
the real benefi t of their HINARI subscription now?
We fear that the loss of access to many key journals that are
published by the major companies could be a major setback
to the education of medical students in Peru and perhaps
around the world. Furthermore, it could make biomedical
research in developing countries like Peru, a key element in
fi ghting poverty, even scarcer.
In conclusion, students and researchers in developing
countries such as Peru, working at the frontlines of global
health problems, need to access more biomedical journals
in order to practice evidence-based health care and conduct
high-quality research. The recent loss of access to many key
biomedical journals in Peru could be a step backwards. We
hope the situation described in this letter might help lend
support to the proposal of Godlee et al., who suggested
that the World Health Organization and its partners should
take the lead in establishing an international collaborative
group along the lines of the Global Fund to fi ght AIDS,
Tuberculosis and Malaria to achieve the goal of “Universal
access to essential health-care information by 2015” or
“Health information for all” [4].

Javier Villafuerte-Gálvez
Walter H. Curioso (wcurioso@u.washington.edu)
Oscar Gayoso
Universidad Peruana Cayetano Heredia
Lima, Peru
References
1. The PLoS Medicine Editors (2006) How can biomedical journals help
to tackle global poverty? PLoS Med 3: e380. doi:10.1371/journal.
pmed.0030380
2. Aronson B (2002) WHO’s Health InterNetwork Access to Research
Initiative (HINARI). Health Info Libr J 19: 164–165.
3. Warschawski DR (2005) Journal impact factors. Available: http://www.ibpc.
fr/~dror/jif.html. Accessed 23 May 2007.
4. Godlee F, Pakenham-Walsh N, Ncayiyana D, Cohen B, Packer A (2004) Can
we achieve health information for all by 2015? Lancet 364: 295–300.
Citation: Villafuerte-Gálvez J, Curioso WH, Gayoso O (2007) Biomedical Journals
and Global Poverty: Is HINARI a step backwards? PLoS Med 4(6): e220. doi:10.1371/
journal.pmed.0040220
Copyright: © 2007 Villafuerte-Gálvez et al. This is an open-access article distributed
under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the
original author and source are credited.
Funding: Preparation of this article was supported in part by a grant from
the Fogarty International Center, United States National Institutes of Health
(5D43TW007551).
Competing Interests: The authors have declared that no competing interests exist.
June 2007 | Volume 4 | Issue 6 | e220