Cluster Wild Bootstrapping for Meta-Analysis.

wildmeta

Typical methods to conduct meta-analysis—pooling effect sizes or analyzing moderating effects with meta-regression—work under the assumption that the effect size estimates are independent. However, primary studies often report multiple estimates of effect sizes. Presence of multiple effect sizes leads to dependence as the estimates within each study are likely correlated (e.g., because the same participants provide multiple outcome scores). The increasingly popular method to handle such dependence, robust variance estimation (RVE), results in inflated Type 1 error rate when the number of studies is small (Hedges, Tipton & Johnson, 2010; Tipton, 2015).

Tipton (2015) and Tipton & Pustejovsky (2015) examined several small sample correction methods. Tipton (2015) recommended CR2 type correction for RVE as well as the use of Satterthwaite degrees of freedom for single coefficient tests. Tipton & Pustejovsky (2015) examined corrections for multiple-contrast hypothesis tests. The authors found that the HTZ test, which is an extension of the CR2 correction method with the Satterthwaite degrees of freedom, controlled Type 1 error rate adequately even when the number of studies was small. However, Joshi, Pustejovsky & Beretvas (2022) showed, through simulations, that the HTZ test can be conservative. The authors examined another method, cluster wild bootstrapping (CWB), that has been studied in the econometrics literature but not in the meta-analytic context. The results of the simulations from Joshi et al. (2022) showed that CWB adequately controlled for Type 1 error rate and provided higher power than the HTZ test, especially for multiple-contrast hypothesis tests.

The goal of this package is to provide applied meta-analytic researchers a set of functions with which they can conduct single coefficient tests or multiple-contrast hypothesis tests using cluster wild bootstrapping.

Installation

Install the latest release from CRAN:

install.packages("wildmeta")

You can install the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("meghapsimatrix/wildmeta")

Example

The following example uses the SATCoaching dataset from the clubSandwich package (Pustejovksy, 2020), originally from DerSimonian and Laird (1983). The standardized mean differences represent the effects of SAT coaching on SAT verbal (SATV) and/or SAT math (SATM) scores.

The code below runs cluster wild bootstrapping to test the multiple-contrast hypothesis that the effect of coaching does not differ based on study type. The study_type variable indicates whether groups compared in primary studies were matched, randomized, or non-equivalent. The meta-regression model also controls for hours of coaching provided (hrs) and whether the students took math or verbal test (test). Below, we run a zero-intercept meta-regression model.

Below, we use the robumeta::robu() function to fit the full model. The functions in our package work with models fit using robumeta::robu() (Fisher, Tipton, & Zhipeng, 2017) and metafor::rma.mv() (Viechtbauer, 2010).

library(wildmeta)
library(clubSandwich)
library(robumeta)

full_model <- robu(d ~ 0 + study_type + hrs + test,
                   studynum = study,
                   var.eff.size = V,
                   small = FALSE,
                   data = SATcoaching)

Wald_test_cwb(full_model = full_model,
              constraints = constrain_equal(1:3),
              R = 12,
              seed = 20201210)
#>   Test Adjustment CR_type Statistic  R     p_val
#> 1  CWB        CR0     CR0   Naive-F 12 0.4166667

Related Work

We want to recognize other packages that provide algorithms to conduct bootstrapping. The fwildclusterboot package runs the fast version of cluster wild bootstrapping proposed by Roodman et al. (2019) and is based on the Stata’s boottest package (Fischer & Roodman, 2021). The package runs bootstrapping for linear regression models and fixed effects models. To the best of our knowledge, meta-analytic models, weighted linear regression models, and multiple contrast hypothesis tests are not supported by the package.

The multiwayvcov package implements cluster robust variance estimation as well as different types of cluster bootstrapping, including pair, residual, and cluster wild bootstrapping (Graham, Arai & Hagströmer, 2016). For cluster wild bootstrapping, Rademacher weights, Mammen weights, and standard normal weights are available. The functions in the package return cluster robust variance-covariance matrices. Our package, on the other hand, outputs p-values from bootstrap hypothesis tests. Further, our package is particularly created for applied meta-analysis whereas the multiwayvcov can be used with any regression analysis involving clusters.

The boot package can be used to generate bootstrap replicates (Canty & Ripley, 2020). The main function in the package, boot() requires a dataset and a function that calculates the estimate of interest. The function returns the bootstrap estimate. The package also contains another function, boot.ci() that can be used to calculate bootstrap confidence intervals. The package allows parallel programming. However, the boot package does not provide functionality for cluster wild bootstrapping.

The lmboot package implements pair, residual and wild bootstrapping for linear models (Heyman, 2019). The package does not provide Rademacher or Mammen weights for wild bootstrapping. The output of the function is a bootstrap sampling distribution. Further, the package does not work with clustered data.

Acknowledgments

Huge shout out to my extremely talented friend RAZ for creating our hex sticker within minutes. We are also extremely thankful to Wolfgang Viechtbauer for helping us solve this issue we had with the update() function. We also thank Mikkel Vembye for testing our package and giving us very helpful feedback, and Sangdon Lim for helping us debug some issues.

References

Canty A. & Ripley B. (2020). boot: Bootstrap R (S-Plus) Functions. R package version 1.3-25. https://CRAN.R-project.org/package=boot

DerSimonian, R., & Laird, N. (1983). Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review, 53(1), 1-1.

Fischer, A. & Roodman, D. (2021). fwildclusterboot: Fast Wild Cluster Bootstrap Inference for Linear Regression Models. Available from https://cran.r-project.org/package=fwildclusterboot.

Fisher, Z., Tipton, E., & Zhipeng, H. (2017). robumeta: Robust variance meta-regression. Retrieved from https://CRAN.R-project.org/package=robumeta

Graham N., Arai M., & Hagströmer, B (2016). multiwayvcov: Multi-Way Standard Error Clustering. R package version 1.2.3. https://CRAN.R-project.org/package=multiwayvcov

Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65.

Heyman, M. (2019). lmboot: Bootstrap in Linear Models. R package version 0.0.1. https://CRAN.R-project.org/package=lmboot

Joshi, M., Pustejovsky, J. E., & Beretvas, S. N. (2022). Cluster wild bootstrapping to handle dependent effect sizes in meta-analysis with small number of studies. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1554

Pustejovsky, J. E. (2020). clubSandwich: Cluster-robust (sandwich) variance estimators with small-sample corrections [R package version 0.4.2]. R package version 0.4.2. https://CRAN.R-project.org/package=clubSandwich

Tipton, E. (2015). Small sample adjustments for robust variance estimation with meta-regression. Psychological Methods, 20(3), 375–393.

Tipton, E., & Pustejovsky, J. E. (2015). Small-Sample Adjustments for Tests of Moderators and Model Fit Using Robust Variance Estimation in Meta-Regression. Journal of Educational and Behavioral Statistics (Vol. 40).

Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48.