TY - JOUR
T1 - Identification of Real and Artifactual Moderators of Effect Size in Meta-Analysis
AU - Collins, Mark
AU - Carey, Timothy
N1 - Funding Information:
The study with the most participants (Keller et al., 2000) was supported by Bristol-Myers Squibb. This was the only study to specifically select participants with chronic major depression. Additionally, it used the HRSD-24 as an outcome measure as opposed to the more commonly used HRSD-17. The next largest study (Browne et al., 2002) was sponsored by the Medical Research Council of Canada in partnership with the Pharmaceutical Manufacturers Association of Canada and Pfizer Canada, Inc. This study differed from most others in that it recruited participants with dysthymia, as opposed to a major depressive diagnosis. Also, it was the only study that used neither the HRSD nor BDI as an outcome measure, but rather the Montgomery-Åsberg Depression Rating Scale (MADRS: Montgomery & Åsberg, 1979). The third largest study, by Miranda et al. (2000), funded by the National Institute of Mental Health (NIMH) and using Paroxetine supplied by GlaxoSmithKline, was the only study to focus exclusively on women from low-income families. Fifty percent of the women in their sample were Black, 44% Latina, and the remainder White. The point of the above comparison is to illustrate the logical difficulty in assigning weights to each of these studies in the statistical analyses that follow. Such weights are traditionally assigned simply on the basis of relative sample sizes (e.g., Shadish & Haddock, 2009), but this rubric assumes that the samples come from the same underlying population. This is clearly not a defensible assumption in this case. In order to avoid having the overall results overly dependent on these three unique studies, results from unweighted analyses will be reported in this article in addition to the customary weighted-by-sample-size analyses. We have adopted this process under the premise that large samples from unique populations may unduly bias the results toward these populations. Furthermore, a recent ecological simulation by Fletcher and Dixon (2011) demonstrated that unweighted linear models will often be more robust than weighted models because they do not make use of potentially poor information on the measurement error variances.
Publisher Copyright:
© 2015, Copyright © Taylor & Francis Group, LLC.
PY - 2015
Y1 - 2015
N2 - This article argues that while meta-analytic studies are widely used in psychological literature, heterogeneity and the potential for confounding remain major problems in the interpretation of meta-analytic study results. The article demonstrates the use of exploratory analysis including graphical methods prior to meta-analysis, and introduces a methodology to screen for artifactual effects. These procedures are illustrated on effect size data comparing depression treatment outcome from psychotherapy versus pharmacotherapy. Results support prior findings of a nonsignificant difference in effect size between the two treatments. They also support findings that treatment type accounts for only a very small proportion of outcome variance. However, the results indicate that some previously reported covariates of depression treatment outcome may be artifactual.
AB - This article argues that while meta-analytic studies are widely used in psychological literature, heterogeneity and the potential for confounding remain major problems in the interpretation of meta-analytic study results. The article demonstrates the use of exploratory analysis including graphical methods prior to meta-analysis, and introduces a methodology to screen for artifactual effects. These procedures are illustrated on effect size data comparing depression treatment outcome from psychotherapy versus pharmacotherapy. Results support prior findings of a nonsignificant difference in effect size between the two treatments. They also support findings that treatment type accounts for only a very small proportion of outcome variance. However, the results indicate that some previously reported covariates of depression treatment outcome may be artifactual.
UR - http://www.scopus.com/inward/record.url?scp=84924067710&partnerID=8YFLogxK
UR - http://www.mendeley.com/research/identification-real-artifactual-moderators-effect-size-metaanalysis
U2 - 10.1080/00273171.2014.963193
DO - 10.1080/00273171.2014.963193
M3 - Article
SN - 0027-3171
VL - 50
SP - 109
EP - 125
JO - Multivariate Behavioral Research
JF - Multivariate Behavioral Research
IS - 1
ER -