Cluster Wild Bootstrapping for Meta-Analysis.
Typical methods to conduct meta-analysis—pooling effect sizes or analyzing moderating effects with meta-regression—work under the assumption that the effect size estimates are independent. However, primary studies often report multiple estimates of effect sizes. Presence of multiple effect sizes leads to dependence as the estimates within each study are likely correlated (e.g., because the same participants provide multiple outcome scores). The increasingly popular method to handle such dependence, robust variance estimation (RVE), results in inflated Type 1 error rate when the number of studies is small (Hedges, Tipton & Johnson, 2010; Tipton, 2015).
Tipton (2015) and Tipton & Pustejovsky (2015) examined several small sample correction methods. Tipton (2015) recommended CR2 type correction for RVE as well as the use of Satterthwaite degrees of freedom for single coefficient tests. Tipton & Pustejovsky (2015) examined corrections for multiple-contrast hypothesis tests. The authors found that the HTZ test, which is an extension of the CR2 correction method with the Satterthwaite degrees of freedom, controlled Type 1 error rate adequately even when the number of studies was small. However, Joshi, Pustejovsky & Beretvas (2022) showed, through simulations, that the HTZ test can be conservative. The authors examined another method, cluster wild bootstrapping (CWB), that has been studied in the econometrics literature but not in the meta-analytic context. The results of the simulations from Joshi et al. (2022) showed that CWB adequately controlled for Type 1 error rate and provided higher power than the HTZ test, especially for multiple-contrast hypothesis tests.
The goal of this package is to provide applied meta-analytic researchers a set of functions with which they can conduct single coefficient tests or multiple-contrast hypothesis tests using cluster wild bootstrapping.
Install the latest release from CRAN:
You can install the development version from GitHub with:
# install.packages("devtools") devtools::install_github("meghapsimatrix/wildmeta")
The following example uses the
SATCoaching dataset from the
clubSandwich package (Pustejovksy, 2020), originally from DerSimonian and Laird (1983). The standardized mean differences represent the effects of SAT coaching on SAT verbal (SATV) and/or SAT math (SATM) scores.
The code below runs cluster wild bootstrapping to test the multiple-contrast hypothesis that the effect of coaching does not differ based on study type. The
study_type variable indicates whether groups compared in primary studies were matched, randomized, or non-equivalent. The meta-regression model also controls for hours of coaching provided (
hrs) and whether the students took math or verbal test (
test). Below, we run a zero-intercept meta-regression model.
Below, we use the
robumeta::robu() function to fit the full model. The functions in our package work with models fit using
robumeta::robu() (Fisher, Tipton, & Zhipeng, 2017) and
metafor::rma.mv() (Viechtbauer, 2010).
library(wildmeta) library(clubSandwich) library(robumeta) full_model <- robu(d ~ 0 + study_type + hrs + test, studynum = study, var.eff.size = V, small = FALSE, data = SATcoaching) Wald_test_cwb(full_model = full_model, constraints = constrain_equal(1:3), R = 12, seed = 20201210) #> Test Adjustment CR_type Statistic R p_val #> 1 CWB CR0 CR0 Naive-F 12 0.4166667
We want to recognize other packages that provide algorithms to conduct bootstrapping. The
fwildclusterboot package runs the fast version of cluster wild bootstrapping proposed by Roodman et al. (2019) and is based on the Stata’s boottest package (Fischer & Roodman, 2021). The package runs bootstrapping for linear regression models and fixed effects models. To the best of our knowledge, meta-analytic models, weighted linear regression models, and multiple contrast hypothesis tests are not supported by the package.
multiwayvcov package implements cluster robust variance estimation as well as different types of cluster bootstrapping, including pair, residual, and cluster wild bootstrapping (Graham, Arai & Hagströmer, 2016). For cluster wild bootstrapping, Rademacher weights, Mammen weights, and standard normal weights are available. The functions in the package return cluster robust variance-covariance matrices. Our package, on the other hand, outputs p-values from bootstrap hypothesis tests. Further, our package is particularly created for applied meta-analysis whereas the
multiwayvcov can be used with any regression analysis involving clusters.
boot package can be used to generate bootstrap replicates (Canty & Ripley, 2020). The main function in the package,
boot() requires a dataset and a function that calculates the estimate of interest. The function returns the bootstrap estimate. The package also contains another function,
boot.ci() that can be used to calculate bootstrap confidence intervals. The package allows parallel programming. However, the
boot package does not provide functionality for cluster wild bootstrapping.
lmboot package implements pair, residual and wild bootstrapping for linear models (Heyman, 2019). The package does not provide Rademacher or Mammen weights for wild bootstrapping. The output of the function is a bootstrap sampling distribution. Further, the package does not work with clustered data.
Huge shout out to my extremely talented friend RAZ for creating our hex sticker within minutes. We are also extremely thankful to Wolfgang Viechtbauer for helping us solve this issue we had with the
update() function. We also thank Mikkel Vembye for testing our package and giving us very helpful feedback, and Sangdon Lim for helping us debug some issues.
Canty A. & Ripley B. (2020). boot: Bootstrap R (S-Plus) Functions. R package version 1.3-25. https://CRAN.R-project.org/package=boot
DerSimonian, R., & Laird, N. (1983). Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 53(1), 1-1.
Fischer, A. & Roodman, D. (2021). fwildclusterboot: Fast Wild Cluster Bootstrap Inference for Linear Regression Models. Available from https://cran.r-project.org/package=fwildclusterboot.
Fisher, Z., Tipton, E., & Zhipeng, H. (2017). robumeta: Robust variance meta-regression. Retrieved from https://CRAN.R-project.org/package=robumeta
Graham N., Arai M., & Hagströmer, B (2016). multiwayvcov: Multi-Way Standard Error Clustering. R package version 1.2.3. https://CRAN.R-project.org/package=multiwayvcov
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.
Heyman, M. (2019). lmboot: Bootstrap in Linear Models. R package version 0.0.1. https://CRAN.R-project.org/package=lmboot
Joshi, M., Pustejovsky, J. E., & Beretvas, S. N. (2022). Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with small number of studies. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1554
Pustejovsky, J. E. (2020). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections [R package version 0.4.2]. R package version 0.4.2. https://CRAN.R-project.org/package=clubSandwich
Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393.
Tipton, E., & Pustejovsky, J. E. (2015). Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression. Journal of Educational and Behavioral Statistics (Vol. 40).
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.