Aims, Design, and Implementation (p. 93-94)
In this part of the book, a comprehensive Monte Carlo study for the comparative evaluation of the statistical approaches will be presented. First, the aims and general procedure will be outlined. Procedural details will be given to enable an assessment of the precision of the study and justify the validity of the results to be presented in Chapter 8. Next, the parameters characterizing the universe from which the effect sizes are drawn will be presented and related to the situations of fixed and random effects as outlined in Chapter 4. This will define the scope of interpretation of the results and shed light on viable generalizations of the results. Finally, technical details on the generation of correlation coefficients in Monte Carlo studies in general are discussed and some specifications for software programming to conduct the Monte Carlo study are given.
Monte Carlo studies are designed to investigate the properties of statistical procedures, techniques, or estimators in particular by conducting a specified number of replications of a statistical procedure when an analytical treatment of the problem is not feasible. In a sense, they can be regarded as experiments conducted to study the behavior of statistics of interests subject to the variation of a set of parameters within the framework of a prespecified model. Accordingly, the design of a Monte Carlo study delimits the scope of interpretation of the results (see Skrondal, 2000). If interest lies, for example, in the performance or robustness of a parameter's estimator, it can only be evaluated with respect to the specific other parameters of the model that have been varied or held constant in a Monte Carlo study. Hence, in the following sections the design of the Monte Carlo study conducted to compare the computational approaches of meta-analysis as outlined in the previous chapters will be described in detail.
7.1 GENERAL AIMS AND PROCEDURE
The main aim of the Monte Carlo study is to compare as well as evaluate the various statistical approaches of meta-analysis as presented in Chapter 5. One of the most important questions to be answered based on the results is whether and when the choice of an approach of meta-analysis makes a difference. In the present Monte Carlo study, the effect sizes under scrutiny will be confined to correlations. Only the d-statistic will be of concern insofar as correlations can be transformed to d and the meta-analysis be based on these transformed effect sizes. The correlation coefficient was chosen as an effect size measure to compare the meta-analytical approaches for several reasons. First, it is one of the most often reported effect sizes indices in the empirical literature in the social sciences and psychology in particular. It therefore represents one of the most representative effect size measures in these scientific areas. Second, all the approaches presented in Chapter 5 explicitly propose procedures to aggregate this effect size measure. Third, its various forms can be easily accommodated to express the size of an effect in a wide variety of research situations and also for results from focused hypothesis tests, a fact that lead several researchers to strongly advocate its use (e.g., Rosenthal & DiMatteo, 2001; Rosenthal et al., 2000). The empirical comparison of meta-analytic approaches is thus limited to a research database consisting of correlations.
If present, differences between the results of these approaches will be highlighted and compared to expectations from an analytical point of view. Comparisons of empirical results with the latter type of expectations are also of interest insofar as many of the theoretical results presented and referenced in Part II hold only asymptotically. Thus, it will be investigated whether the application of the proposed procedures yields sufficiently accur