7 Meta-Analysis of Randomized Clinical Trials in the Evaluation of Medical Treatments - A Partly Regulatory Perspective (p. 99-100)
Meta-analysis of randomized clinical trials plays an important role in summarizing available evidence with respect to the comparison of different drugs for the same indication. In contrast, up to now meta-analysis is of only minor importance in the process of new drug application despite the fact that also in this situation a summary evaluation of available evidence from a, although limited, number of independent clinical trials is necessary. The main reason is, in our opinion, that presented meta-analyses often are not completely convincing because objectives are not appropriately chosen and conduct or presentation are not sufficiently detailed so that the reader can assess provided evidence. This chapter is intended to clarify why some meta-analyses have higher credibility than others and provide some guidance to how the credibility of meta-analyses can be increased. Presented ideas are, hopefully, not only in the regulatory setting of importance.
Meta-analysis has been defined to be a quantitative and systematic summary of a collection of separate studies for the purpose of obtaining information that can not be derived from any of the studies alone (Boissel et al., 1988). With this definition, meta-analysis implicitly is also a technique that should lead to reproducible results and that can be distinguished from the classical review or overview, where results from various studies might be collected and qualitatively weighted by an expert in the field.
Originally invented in the social sciences, meta-analysis has found widespread use in clinical research during the last two decades and the per-year number of published meta-analysis is still increasing. However, only in rare cases has the discussion about the appropriateness of biostatistical methodology in medical research been as intensive as was the case with meta-analysis. From the very beginning meta-analysis has split up the community into clear proponents and those who completely dislike this type of analysis.
Feinstein (1995) named meta-analysis a synonym for "statistical alchemy for the 21st century", and others expressed their doubts on the credibility of results "proven" by means of meta-analysis. It has repeatedly been emphasized that pivotal trials should be designed to stand on their own and that in consequence meta-analyses should not be necessary ("If a treatment has an effect so recondite and obscure as to require meta-analysis to establish it I would not be happy to have it used on me" (Eysenck, 1994, p. 792)). And also empirical comparisons of the results from meta-analyses with results from large randomized clinical trials (Villar, Carroli, & Belizan, 1995) or critical expert reading of meta-analyses do not support the hypothesis that a meta-analysis can replace randomized clinical trials ("In my own review of selected metaanalyses, problems were so frequent and so serious, including bias on part of the meta-analyst, that it was difficult to trust in the overall 'best estimates' that the method often produces" (Bailar, 1997, p. 560)).
A positive view on meta-analysis is best summarized by a citation from a recent paper by Resch (1996), who wrote:
I disagree, however, that a meta-analysis should exclusively be viewed as "hypothesis generating". This proposal denies the fact that, however biased, a high-quality meta-analysis quantitatively summarizes the existing evidence. What could be a better basis for a clinician's treatment decision at the time it must be made? (p. 621)
Meta-analyses - being retrospective and non-experimental investigations - are in a strict sense observational studies (Victor, 1995). Comparing, however, the evidence gained from a prospective obse