knitr::opts_chunk$set(
    message = FALSE,
    warning = FALSE,
    include = TRUE,
    error = FALSE, 
    fig.width=7, 
    fig.height=5
)




library(groundhog)
groundhog.library(c("tidyverse", "kableExtra", "MAd", "meta"), date = "2024-02-14")

windowsFonts(Times = windowsFont("Times New Roman"))
options(scipen = 999, digits = 3)
theme_set(theme_minimal(base_size = 12, base_family = "Times"))

1 Meta-analysis 1

MA1 <- readRDS("data/preprocessed/MA1.rds")
  
# average group sample size
mean((MA1$nc_ma + MA1$nt_ma) / 2)
## [1] 13

1.1 Reproduce primary SMDs

Between-groups studies which reported sample sizes, means and SDs (SEs/CIs)

MA1 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd") %>%
  mutate(d = abs(d)) %>% 
  # relocate to facilitate inspection
  relocate(c(id_comparison, d, smd_ma), .after = dv_ps)

The SMDs that could not be reproduced or approximated (those based on values from figures can only be approximated due to variations in how values were extracted from figures): (4/17 comparisons from between-groups studies). Reproduced SMDs 16, 17, 18, and 20 deviate from the reported ones beyond what would be expected as a result of rounding or differences in how values were extracted from figures. For SMDs 16 and 20, other values could have been used:

For SMD 16, another set of means and SDs could have been used:

compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$d[[16]]
## [1] 1.43

Reproduces reported SMD successfully.

SMD 20 had corresponding \(p\)-value:

p_to_d1(MA1$p_ps[[20]], MA1$nt_ps[[20]], MA1$nc_ps[[20]])[1]
## [1] 0.672
p_to_d2(MA1$p_ps[[20]], MA1$nt_ps[[20]], MA1$nc_ps[[20]])[1]
## [1] 0.718

The SMD based on the two tailed \(p\) value successfully reproduces the reported SMD.

Crossover studies which reported sample sizes, means and SDs (SEs/CIs)

Since the meta-analysts listed the sample size of both the treatment and control group as = \(N\) = the number of participants, I use N as the sample size of both groups even though there aren’t two groups:

MA1_co <- MA1 %>% filter(is.na(nc_ps))
MA1_co <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1_co, denom = "pooled.sd")

Reproduced SMDs 12, 13, 21 deviate from the reported ones beyond what would be expected as a result of rounding or differences in how values were extracted from figures. For SMDs 12 and 13, other values could have been used: SMDs 12 and 13 and 21 had another set of means and SDs:

compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$d[[12]]
## [1] 1.63
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$d[[13]]
## [1] 1.61
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$d[[21]]
## [1] 1.41

Still not reproducible. SMD 13 does reproduce the Cohen’s \(d\) reported in the primary study, though.

1.2 Overview of SMDs post double checking irreproducible SMDs

Data extraction was correct for all SMDs and no further data were extracted (see data sheets column “comments” for sources of data).

  1. Faithfully reproducible
  2. Faithfully approximated by entering the total sample size as both the control and treatment groups sample size.
  3. Faithfully reproducible
  4. Faithfully approximated
  5. Faithfully reproducible
  6. Faithfully reproducible
  7. Faithfully reproducible
  8. Faithfully irreproducible
  9. Faithfully irreproducible
  10. Faithfully approximated
  11. Faithfully approximated
  12. irreproducible. SMD based on the first set of means and SDs will be used for MA model 2 (according to definition below) since the control values here are the baseline (0) values which corresponds more or less to the meta-analysts’ description of their procedure “baseline: Ctrl”.
  13. irreproducible. SMD based on the first set will be used for MA model 2 since the control values here are the baseline (0) values which corresponds more or less to the meta-analysts’ description of their procedure “baseline: Ctrl”.
  14. Faithfully reproducible
  15. Brute-force approximated because based on change from baseline values when the MA gives “Score in JHFT (atDCS on iH during OT at retention: Rx vs sham during OT at retention: Ctrl)” as outcome
  16. Faithfully irreproducible using the values purported to have been used by the meta-analysts “3J-ROM(ctDCS on cH during OT at retention: Rx vs sham during OT at retention: Ctrl)”, successfully reproduced using “Proportional change in 3J-ROM” values. The SMD based on the latter values will be used for MA model 2 .
  17. Faithfully irreproducible.
  18. Faithfully irreproducible.
  19. Reproducible but not according to the information given in the MA. the values used are for changes in JHFT time from baseline whereas the meta-analysts report using “Time in JHFT”.
  20. Brute-force reproducible using a \(p\)-value based on difference in medians test.
  21. irreproducible.

3 meta-analytic models were fit to each meta-analysis:

  1. MAM1: Using the SMDs reproduced strictly following information given in the meta-analysis or a standard procedure apparently adopted for at least 2 other primary SMDs (e.g., Cohen’s \(d\) of tDCS vs. Sham at post intervention).
  2. MAM2: Using the SMDs reproduced strictly following information given in the meta-analysis or a standard procedure apparently adopted for at least 2 other primary SMDs + SMDs that could be successfully reproduced (i.e., reproduced SMD = reported SMD) following a procedure which either does not entirely correspond to what the meta-analysts report having adopted or does not (necessarily) produce an SMD that is comparable to what would result from following the procedure apparently adopted for at least 2 other primary SMDs
  3. MAM3: Using the reported SMDs only (reproduce fitting the MA model using values reported in the MA only)

1.3 MAM1 collation

1.3.1 SMDs

# faithfully reproduced SMDs
MA1$smd_re1 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d
MA1$smd_re1[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[2]]
MA1$smd_re1[[21]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[21]]

# Brute-force reproduced SMDs

MA1$smd_re1[[19]] <- NA

MA1$smd_re1[[20]] <- NA

MA1$smd_re1[[12]] <- NA

MA1$smd_re1[[13]] <- NA
  
# make numeric and positive
MA1$smd_re1 <- MA1$smd_re1 %>% as.numeric() %>% abs()

# relocate to check if everything's fine
MA1 <- MA1 %>% relocate(c(id_comparison, smd_re1, smd_ma), .after = dv_ps)

1.3.2 sampling variances

MAd’s compute_ds() and p_to_d1 functions automatically calculate the corresponding sampling/sampling variances. In cases where not enough data were available, the square of the standard error extracted from the funnel plot in the MA will be used.

# SMDs which were successfully reproduced using the first set of means and SDs, in accordance with information in MA
MA1$v_re1 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d

MA1$v_re1[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[2]]

# SMDs which could not be reproduced using the first set of means and SDs, in accordance with information in MA
MA1$v_re1[[21]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[21]]

# SMDs which were successfully reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs
# MA1$v_re1[[16]] <- NA

MA1$v_re1[[19]] <- NA

MA1$v_re1[[20]] <- NA

# SMDs which could not be reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs
MA1$v_re1[[12]] <- NA

MA1$v_re1[[13]] <- NA
  
# make numeric
MA1$v_re1 <- as.numeric(MA1$v_re1)

# relocate to check if everything's fine
MA1$v_ma <- MA1$se_ma^2
MA1 <- MA1 %>% relocate(c(id_comparison, v_re1, v_ma), .after = dv_ps)

1.4 MAM2 collation

1.4.1 SMDs

# SMDs which were successfully reproduced using the first set of means and SDs, in accordance with information in MA
MA1$smd_re2 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d



# SMDs which could not be reproduced using the first set of means and SDs, in accordance with information in MA
MA1$smd_re2[[21]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[21]]

# SMDs which were successfully reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs

MA1$smd_re2[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[2]]


MA1$smd_re2[[16]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$d[[16]]

MA1$smd_re2[[19]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[19]]

MA1$smd_re2[[20]] <- p_to_d2(MA1$p_ps[[20]], MA1$nt_ps[[20]], MA1$nc_ps[[20]])[1]

# SMDs which could not be reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs
MA1$smd_re2[[12]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[12]]

MA1$smd_re2[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$d[[13]]
  
# make numeric and positive
MA1$smd_re2 <- MA1$smd_re2 %>% as.numeric() %>% abs()

# relocate to check if everything's fine
MA1 <- MA1 %>% relocate(smd_re2, .after = smd_re1)

1.4.2 sampling variances

# SMDs which were successfully reproduced using the first set of means and SDs, in accordance with information in MA
MA1$v_re2 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d

MA1$v_re2[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[2]]

# SMDs which could not be reproduced using the first set of means and SDs, in accordance with information in MA
MA1$v_re2[[21]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[21]]

# SMDs which were successfully reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs
MA1$v_re2[[16]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA1, denom = "pooled.sd")$var.d[[16]]

MA1$v_re2[[19]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[19]]

MA1$v_re2[[20]] <- p_to_d2(MA1$p_ps[[20]], MA1$nt_ps[[20]], MA1$nc_ps[[20]])[2]

# SMDs which could not be reproduced using values that either do not quite correspond to the information given in the MA or the standard method adopted for the other SMDs
MA1$v_re2[[12]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[12]]

MA1$v_re2[[13]] <- MA1$smd_re2[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA1, denom = "pooled.sd")$var.d[[13]]

  
# make numeric
MA1$v_re2 <- as.numeric(MA1$v_re2)

# relocate to check if everything's fine
MA1$v_ma <- MA1$se_ma^2
MA1 <- MA1 %>% relocate(c(id_comparison, v_re2, v_ma), .after = dv_ps)

1.5 Reproduce the pooled SMD

1.5.1 MA model 1

mam1 <- metagen(TE = smd_re1,
              seTE = sqrt(v_re1),
              studlab = id_comparison,
              data = MA1,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )

forest(mam1,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

1.5.2 MA model 2

mam2 <- metagen(TE = smd_re2,
              seTE = sqrt(v_re2),
              studlab = id_comparison,
              data = MA1,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )

forest(mam2,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

1.5.3 MA model 3

mam3 <- metagen(TE = smd_ma,
              seTE = se_ma,
              studlab = id_comparison,
              data = MA1,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )

forest(mam3,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

2 Meta-analysis 2

MA2 <- readRDS("data/preprocessed/MA2.rds")

# average group sample size
mean((MA2$nc_ma + MA2$nt_ma) / 2)
## [1] 12.2

2.1 Reproduce primary SMDs

Between-groups studies which reported sample sizes, means and SDs (SEs/CIs)

#MA2 <- compute_dgs(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2, denom = "pooled.sd")
# relocate to facilitate inspection
#MA2 <- MA2 %>% relocate(c(d, g, smd_ma), .after = dv_ps)

For PSs that reported means and SDs (no extraction from figure necessary), all reported SMDs were approximated better by Cohen’s \(d\)s than Hedges’ \(g\)s. I will thus assume the authors used Cohen’s \(d\)s:

MA2 <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2, denom = "pooled.sd")

# relocate to facilitate inspection
MA2 <- MA2 %>% relocate(c(id_comparison, d, smd_ma), .after = dv_ps)

Reproduced SMDs 6, 7, 9, 12, 14, 15 deviate from the reported ones beyond what would be expected as a result of rounding or differences in how values were extracted from figures. For SMDs 6, 7, 14, and 15, other values could have reasonably been used:

SMDs 6 and 7 had corresponding \(p\)-values:

# one-tailed p-value + sample sizes to d
p_to_d1(0.925, 6, 5)[1]
## [1] -0.953
p_to_d1(0.897, 6, 5)[1]
## [1] -0.825
# two-tailed p-value + sample sizes to d
p_to_d2(0.925, 6, 5)[1]
## [1] 0.0586
p_to_d2(0.897, 6, 5)[1]
## [1] 0.0806

Since these are even further from the ones based on means and SDs, data extraction was double checked for correctness (see below).

For SMDs 14 and 15, two other sets of means and SDs could have been used:

# Set 2
compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[14]]
## [1] 1.84
compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[15]]
## [1] 1.16
# Set 3
compute_ds(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA2, denom = "pooled.sd")$d[[14]]
## [1] 1.43
compute_ds(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA2, denom = "pooled.sd")$d[[15]]
## [1] 1.01

Values of set 2 are much closer, but not quite identical.

Crossover studies which reported sample sizes, means and SDs (SEs/CIs) Since the meta-analysts listed the sample size of both the treatment and control group as = \(N\) = the number of participants, I use N as the sample size of both groups even though there aren’t two groups:

MA22 <- MA2 %>% filter(is.na(nc_ps))
MA22 <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA22, denom = "pooled.sd")

Reproduced SMDs 1, 2, 4, 5, 13, 20 deviate from the reported ones beyond what would be expected as a result of rounding or differences in how values were extracted from figures. For SMDs 1, 2, 4, 5, and 20, other values could have reasonably been used:

SMD 1 had two other sets of means and SDs and a corresponding \(p\) value:

# Set 2
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[1]]
## [1] 0
# Set 3
compute_ds(n_ps, mt_ps.3, sdt_ps.3, n_ps, mc_ps.3, sdc_ps.3, data = MA2, denom = "pooled.sd")$d[[1]]
## [1] -0.38
# p-value
p_to_d1(MA2$p_ps[[1]], MA2$n_ps[[1]], MA2$n_ps[[1]])[1]
## [1] 1.07
p_to_d2(MA2$p_ps[[1]], MA2$n_ps[[1]], MA2$n_ps[[1]])[1]
## [1] 1.22

The \(d\) based on the one-tailed \(p\)-values is closest.

SMD 2 had a corresponding \(p\) value:

p_to_d1(MA2$p_ps[[2]], MA2$n_ps[[2]], MA2$n_ps[[2]])[1]
## [1] 1.97
p_to_d2(MA2$p_ps[[2]], MA2$n_ps[[2]], MA2$n_ps[[2]])[1]
## [1] 2.35

Closer.

SMDs 4, 5, and 20 had another set of means and SDs:

compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[4]]
## [1] -0.542
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[5]]
## [1] 0.378
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[20]]
## [1] 1.41

Nope.

Studies which did not report means and/or SDs (SEs/CIs) For SMD 8, only means were reported. Computing an SMD is thus not possible. For SMD 19 no means or SDs were reported but a potentially suitable \(p\)-value:

p_to_d1(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[1]
## [1] 0.946
p_to_d2(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[1]
## [1] 1.11

The one-tailed one is close enough.

For SMD 3, an SMD was reported in the primary study which was close to the one reported in the MA.

2.2 Overview of SMDs post double checking irreproducible SMDs

  1. Data extraction correct, primary SMD irreproducible. The SMD based on the \(p\)-value is closer to the reported one but will not be used for reproducing the pooled SMD because the \(p\)-value is for the comparison tDCS_post vs. tDCS_pre and not tDCS_post vs. sham_post (which is supposed to be the outcome). The SMD based on the second set of means and SDs will be used because it’s closest to the reported one even though the means and SDs are raw and not “change in PPT”.
  2. Data extraction correct, primary SMD irreproducible. The SMD based on the \(p\)-value is closer to the reported one but will not be used for reproducing the pooled SMD because the \(p\)-value is for the comparison in the improvement from baseline between tDCS condition, not “JTT (sham vs. ctDCS at post)” (which is supposed to be the outcome). The reproduced SMD based on the means and SDs will be used for the pooled SMD.
  3. The SMD reported in the primary study will be used to reproduce the pooled SMD. The SMD is for improvement of tDCS group from baseline ==> model 2
  4. Data extraction correct, primary SMD irreproducible. The SMD based on the first set of means and SDs will be used because it’s the percentage of change means and SDs, which is supposed to be the outcome used in the MA.
  5. Data extraction correct, primary SMD irreproducible. The SMD based on the second set of means and SDs will be used because they are derived by separating the N = 9 into two groups n = 6 and 3, respectively, which is how the meta-analysts report having done it.
  6. Since the means and SDs from the first set are for changes in the outcome and not raw, I extracted raw means and SDs from figures. SMDs based on these values will be used for the pooled SMD. The SMD is approximated
  7. Since the means and SDs from the first set are for changes in the outcome and not raw, I extracted raw means and SDs from figures. SMDs based on these values will be used for the pooled SMD. The SMD is irreproducible.
  8. successfully reproduced using a \(p\) value for the comparison between the 3 groups cathodal, anodal and sham at post, will be used for MA model 2
  9. Data extraction correct, primary SMD irreproducible. successfully reproduced using a \(p\) value for the comparison between the 3 groups cathodal, anodal and sham at post, will be used for MA model 2
  10. SMD can be considered as reproducible since the difference is probably due to variation in how the means and SDs were extracted from the figure.
  11. Same as 10, but negative since the scale used is negatively coded.
  12. I made a mistake in extracting sample sizes. The SMD remains irreproducible after correction.
  13. Data extraction correct, primary SMD irreproducible.
  14. Data extraction correct, primary SMD irreproducible. SMDs based on set 2 are closest to the reported SMDs but these are follow up values not post. Thus the values of the first set will be used for reproducing the pooled SMD.
  15. Data extraction correct, primary SMD irreproducible. SMDs based on set 2 are closed to the reported SMDs but these are follow up values not post. Thus the values of the first set will be used for reproducing the pooled SMD.
  16. Data extraction correct, primary SMD irreproducible. Almost the same as the reported SMD but negative.
  17. Reproducible.
  18. Same as 10, but negative since the variable is reaction time.
  19. Approximated by using a \(p\) value for tDCS vs. sham post comparison. Cathodal had a negative effect.
  20. Data extraction correct, primary SMD irreproducible. SMD based on the second set of means and SDs will be used since it’s the post values.

Reproduce SMDs 6 & 7 using values extracted from figures:

# Convert upper bounds of SD bars extracted from figures to SDs
MA2[6, "sdc_ps.2"] <- (MA2[6, "sdc_ps.2"] - MA2[6, "mc_ps.2"])
MA2[6, "sdt_ps.2"] <- (MA2[6, "sdt_ps.2"] - MA2[6, "mt_ps.2"])

MA2[7, "sdc_ps.2"] <- (MA2[7, "sdc_ps.2"] - MA2[7, "mc_ps.2"])
MA2[7, "sdt_ps.2"] <- (MA2[7, "sdt_ps.2"] - MA2[7, "mt_ps.2"])

# Compute Cohen's ds
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[6]]
## [1] 0.412
compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2, denom = "pooled.sd")$d[[7]]
## [1] 0.355

Reproduce SMDs 8 & 9 using \(p\)-values:

p_to_d2(MA2$p_ps[[8]], MA2$nc_ps[[8]], MA2$nt_ps[[8]])[1]
## [1] 0.136
p_to_d2(MA2$p_ps[[9]], MA2$nc_ps[[9]], MA2$nt_ps[[9]])[1]
## [1] 0.109

2.3 MAM1 collation

2.3.1 SMDs

MA2$smd_re1[[1]] <- NA

MA2$smd_re1[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[2,], denom = "pooled.sd")$d

MA2$smd_re1[[3]] <- NA

MA2$smd_re1[[4]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[4,], denom = "pooled.sd")$d

MA2$smd_re1[[5]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[5,], denom = "pooled.sd")$d

MA2$smd_re1[[6]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[6,], denom = "pooled.sd")$d

MA2$smd_re1[[7]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[7,], denom = "pooled.sd")$d

MA2$smd_re1[[8]] <- NA

MA2$smd_re1[[9]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[9,], denom = "pooled.sd")$d

MA2$smd_re1[[10]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[10,], denom = "pooled.sd")$d

MA2$smd_re1[[11]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[11,], denom = "pooled.sd")$d %>% abs()

MA2$smd_re1[[12]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[12,], denom = "pooled.sd")$d

MA2$smd_re1[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[13,], denom = "pooled.sd")$d

MA2$smd_re1[[14]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[14,], denom = "pooled.sd")$d

MA2$smd_re1[[15]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[15,], denom = "pooled.sd")$d

MA2$smd_re1[[16]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[16,], denom = "pooled.sd")$d

MA2$smd_re1[[17]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[17,], denom = "pooled.sd")$d

MA2$smd_re1[[18]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[18,], denom = "pooled.sd")$d %>% abs()

MA2$smd_re1[[19]] <- -p_to_d1(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[1]

MA2$smd_re1[[20]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[20,], denom = "pooled.sd")$d

# make numeric
MA2$smd_re1 <- MA2$smd_re1 %>% as.numeric()

# relocate to check if everything's fine
MA2 <- MA2 %>% relocate(c(id_comparison, smd_re1, smd_ma), .after = dv_ps)

2.3.2 Sampling variances

MA2$v_re1[[1]] <- NA

MA2$v_re1[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[2,], denom = "pooled.sd")$var.d

MA2$v_re1[[3]] <- NA

MA2$v_re1[[4]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[4,], denom = "pooled.sd")$var.d

MA2$v_re1[[5]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[5,], denom = "pooled.sd")$var.d

MA2$v_re1[[6]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[6,], denom = "pooled.sd")$var.d

MA2$v_re1[[7]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[7,], denom = "pooled.sd")$var.d

MA2$v_re1[[8]] <- NA

MA2$v_re1[[9]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[9,], denom = "pooled.sd")$var.d

MA2$v_re1[[10]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[10,], denom = "pooled.sd")$var.d

MA2$v_re1[[11]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[11,], denom = "pooled.sd")$var.d

MA2$v_re1[[12]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[12,], denom = "pooled.sd")$var.d

MA2$v_re1[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[13,], denom = "pooled.sd")$var.d

MA2$v_re1[[14]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[14,], denom = "pooled.sd")$var.d

MA2$v_re1[[15]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[15,], denom = "pooled.sd")$var.d

MA2$v_re1[[16]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[16,], denom = "pooled.sd")$var.d

MA2$v_re1[[17]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[17,], denom = "pooled.sd")$var.d

MA2$v_re1[[18]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[18,], denom = "pooled.sd")$var.d

MA2$v_re1[[19]] <- p_to_d1(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[2]

MA2$v_re1[[20]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[20,], denom = "pooled.sd")$var.d

# make numeric
MA2$v_re1 <- MA2$v_re1 %>% as.numeric()

# relocate to check if everything's fine
MA2$v_ma <- MA2$se_ma^2

MA2 <- MA2 %>% relocate(c(id_comparison, v_re1, v_ma), .after = dv_ps)

2.4 MAM2 collation

2.4.1 SMDs

MA2$smd_re2[[1]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[1,], denom = "pooled.sd")$d

MA2$smd_re2[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[2,], denom = "pooled.sd")$d

MA2$smd_re2[[3]] <- MA2$smd_ps[[3]]

MA2$smd_re2[[4]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[4,], denom = "pooled.sd")$d

MA2$smd_re2[[5]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[5,], denom = "pooled.sd")$d

MA2$smd_re2[[6]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[6,], denom = "pooled.sd")$d

MA2$smd_re2[[7]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[7,], denom = "pooled.sd")$d

MA2$smd_re2[[8]] <- -p_to_d2(MA2$p_ps[[8]], MA2$nc_ps[[8]], MA2$nt_ps[[8]])[1]

MA2$smd_re2[[9]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[9,], denom = "pooled.sd")$d

MA2$smd_re2[[10]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[10,], denom = "pooled.sd")$d

MA2$smd_re2[[11]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[11,], denom = "pooled.sd")$d %>% abs()

MA2$smd_re2[[12]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[12,], denom = "pooled.sd")$d

MA2$smd_re2[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[13,], denom = "pooled.sd")$d

MA2$smd_re2[[14]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[14,], denom = "pooled.sd")$d

MA2$smd_re2[[15]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[15,], denom = "pooled.sd")$d

MA2$smd_re2[[16]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[16,], denom = "pooled.sd")$d

MA2$smd_re2[[17]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[17,], denom = "pooled.sd")$d

MA2$smd_re2[[18]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[18,], denom = "pooled.sd")$d %>% abs()

MA2$smd_re2[[19]] <- -p_to_d1(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[1]

MA2$smd_re2[[20]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[20,], denom = "pooled.sd")$d

# make numeric
MA2$smd_re2 <- MA2$smd_re2 %>% as.numeric()

# relocate to check if everything's fine
MA2 <- MA2 %>% relocate(smd_re2, .after = smd_re1)

2.4.2 Sampling variances

MA2$v_re2[[1]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[1,], denom = "pooled.sd")$var.d

MA2$v_re2[[2]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[2,], denom = "pooled.sd")$var.d

MA2$v_re2[[3]] <- ((MA2$n_ps[[3]] + MA2$n_ps[[3]]) / (MA2$n_ps[[3]]*MA2$n_ps[[3]])) + (MA2$smd_ps[[3]]^2 / (2 * (MA2$n_ps[[3]] + MA2$n_ps[[3]])))

MA2$v_re2[[4]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[4,], denom = "pooled.sd")$var.d

MA2$v_re2[[5]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[5,], denom = "pooled.sd")$var.d

MA2$v_re2[[6]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[6,], denom = "pooled.sd")$var.d

MA2$v_re2[[7]] <- compute_ds(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA2[7,], denom = "pooled.sd")$var.d

MA2$v_re2[[8]] <- p_to_d2(MA2$p_ps[[8]], MA2$nc_ps[[8]], MA2$nt_ps[[8]])[2]

MA2$v_re2[[9]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[9,], denom = "pooled.sd")$var.d

MA2$v_re2[[10]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[10,], denom = "pooled.sd")$var.d

MA2$v_re2[[11]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[11,], denom = "pooled.sd")$var.d

MA2$v_re2[[12]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[12,], denom = "pooled.sd")$var.d

MA2$v_re2[[13]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[13,], denom = "pooled.sd")$var.d

MA2$v_re2[[14]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[14,], denom = "pooled.sd")$var.d

MA2$v_re2[[15]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[15,], denom = "pooled.sd")$var.d

MA2$v_re2[[16]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[16,], denom = "pooled.sd")$var.d

MA2$v_re2[[17]] <- compute_ds(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA2[17,], denom = "pooled.sd")$var.d

MA2$v_re2[[18]] <- compute_ds(n_ps, mt_ps.1, sdt_ps.1, n_ps, mc_ps.1, sdc_ps.1, data = MA2[18,], denom = "pooled.sd")$var.d

MA2$v_re2[[19]] <- p_to_d1(MA2$p_ps[[19]], MA2$n_ps[[19]], MA2$n_ps[[19]])[2]

MA2$v_re2[[20]] <- compute_ds(n_ps, mt_ps.2, sdt_ps.2, n_ps, mc_ps.2, sdc_ps.2, data = MA2[20,], denom = "pooled.sd")$var.d

# make numeric
MA2$v_re2 <- MA2$v_re2 %>% as.numeric()

2.5 Reproduce the pooled SMD

2.5.1 MA model 1

mam1 <- metagen(TE = smd_re1,
              seTE = sqrt(v_re1),
              studlab = id_comparison,
              data = MA2,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )



forest(mam1,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

2.5.2 MA model 2

mam2 <- metagen(TE = smd_re2,
              seTE = sqrt(v_re2),
              studlab = id_comparison,
              data = MA2,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )


forest(mam2,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

2.5.3 MA model 3

mam3 <- metagen(TE = smd_ma,
              seTE = se_ma,
              studlab = id_comparison,
              data = MA2,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )


forest(mam3,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "d", "SE"))

3 Meta-analysis 3

MA3 <- readRDS("data/preprocessed/MA3.rds")

# average group sample size
mean((MA3$nc_ma + MA3$nt_ma) / 2, na.rm = TRUE)
## [1] 16.3

3.1 Reproduce primary SMDs

This MA only included RCTs. 1 out of the 6 studies included was a conference abstract which only reported the total sample size. The 5 others reported group sample sizes, 3 reported raw means and SDs, one reported a potentially usable \(t\)-value. Hedges’ \(g\) was used. Studies which reported means and SDs

MA3 <- compute_dgs(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3, denom = "pooled.sd")

# relocate to facilitate inspection
MA3 <- MA3 %>% relocate(c(id_comparison, g, smd_ma), .after = dv_ps)

SMD 4 was reproduced. SMD 2 was approximated but in the other direction. SMD 5 quite close. Data extraction will be double-checked.

Studies which did not report means and/or SDs

t_to_d(MA3$t_ps[[6]], MA3$nt_ps[[6]], MA3$nc_ps[[6]])[1]
## [1] 0.597

Nope.

3.2 Overview of SMDs post double checking irreproducible SMDs

MA3$d <- p_to_d2(0.01, MA3$nt_ps[[1]], MA3$nc_ps[[1]])[1]
MA3$vd <- p_to_d2(0.01, MA3$nt_ps[[1]], MA3$nc_ps[[1]])[2]
compute_gs(d, vd, nt_ps, nc_ps, MA3)$g[1]
## [1] 0.84
compute_dgs(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.243
compute_dgs(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.308
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2

compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.275
MA3$mt_ps.3[[2]] <- (MA3$mt_ps.1[[2]] + MA3$mt_ps.2[[2]]) / 2
MA3$sdt_ps.3[[2]] <- (MA3$sdt_ps.1[[2]] + MA3$sdt_ps.2[[2]]) / 2

compute_dgs(nt_ps*2, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.289
compute_dgs(nt_ps*2, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.253
compute_dgs(nt_ps*2, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA3[2,], denom = "pooled.sd")$g
## [1] -0.329
MA3$d <- p_to_d2(0.01, MA3$n_ps[[3]], MA3$n_ps[[3]])[1]
MA3$vd <- p_to_d2(0.01, MA3$n_ps[[3]], MA3$n_ps[[3]])[2]
compute_gs(d, vd, n_ps, n_ps, MA3)$g[3]
## [1] 0.982

5.based on second set

compute_dgs(nt_ps, mt_ps.2, sdt_ps.2, nc_ps, mc_ps.2, sdc_ps.2, data = MA3[5,], denom = "pooled.sd")$g
## [1] 0.396

nope

take average

MA3$mc_ps.3 <- (MA3$mc_ps.1 + MA3$mc_ps.2) / 2
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdc_ps.3 <- (MA3$sdc_ps.1 + MA3$sdc_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA3[5,], denom = "pooled.sd")$g
## [1] 0.565

yup

MA3$d <- p_to_d2(0.169, MA3$nc_ps[[6]], MA3$nt_ps[[6]])[1]
MA3$vd <- p_to_d2(0.169, MA3$nc_ps[[6]], MA3$nt_ps[[6]])[2]
compute_gs(d, vd, nt_ps, nc_ps, MA3)$g[6]
## [1] 0.585

yup

  1. SMD 1 could be brute-force reproduced using a \(p\) value (given as a range <0.01) for the comparison between the control and treatment medians at post.
  2. Approximated by taking the first set of means and SDs (SMA) and doubling the sample size for the treatment group (USED FOR MMA2). It was negative because the variable is reaction times. The variation without doubling the sample size and by averaging the two sets will be used for MMA 1 since it corresponds to other reproducible SMDs.
  3. Same as 1
  4. reproducible based on means and SDs of change percentages from pre to post
  5. SMD 5 was approximated well by averaging sets 1 and 2 and computing the SMD based on the averaged values. Faithfully approximated because both outcomes are defined as primary outcomes in the primary study.
  6. SMD 6 could be reproduced using a \(p\) value for the comparison between the control and treatment mean change from baseline.

3.3 MAM1 collation

3.3.1 SMDs

# 1
MA3$smd_re1[[1]] <- NA

# 2

MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
MA3$smd_re1[[2]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g %>% abs()

# 3
MA3$smd_re1[[3]] <- NA

# 4
MA3$smd_re1[[4]] <- NA

#5 
MA3$mc_ps.3 <- (MA3$mc_ps.1 + MA3$mc_ps.2) / 2
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdc_ps.3 <- (MA3$sdc_ps.1 + MA3$sdc_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
MA3$smd_re1[[5]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA3[5,], denom = "pooled.sd")$g

#6
MA3$smd_re1[[6]] <- NA

# make numeric
MA3 <- MA3 %>% mutate_at(vars(starts_with('smd')), as.numeric)

# relocate to facilitate inspection
MA3 <- MA3 %>% relocate(c(smd_re1, smd_ma, id_comparison),  .after = dv_ps)

3.3.2 Sampling variances

# 1
MA3$v_re1[[1]] <- NA

# 2
MA3$v_re1[[2]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$var.g

# 3
MA3$v_re1[[3]] <- NA

# 4
MA3$v_re1[[4]] <- NA

#5 
MA3$mc_ps.3 <- (MA3$mc_ps.1 + MA3$mc_ps.2) / 2
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdc_ps.3 <- (MA3$sdc_ps.1 + MA3$sdc_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
MA3$v_re1[[5]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA3[5,], denom = "pooled.sd")$var.g

#6
MA3$v_re1[[6]] <- NA

3.4 MAM2 collation

3.4.1 SMDs

# 1
MA3$d <- p_to_d2(MA3$p_ps[[1]], MA3$nt_ps[[1]], MA3$nc_ps[[1]])[1]
MA3$vd <- p_to_d2(MA3$p_ps[[1]], MA3$nt_ps[[1]], MA3$nc_ps[[1]])[2]
MA3$smd_re2[[1]] <- compute_gs(d, vd, nt_ps, nc_ps, MA3)$g[1]

# 2
MA3$smd_re2[[2]] <- compute_dgs(nt_ps*2, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$g %>% abs()

# 3
MA3$d <- p_to_d2(MA3$p_ps[[3]], MA3$n_ps[[3]], MA3$n_ps[[3]])[1]
MA3$vd <- p_to_d2(MA3$p_ps[[3]], MA3$n_ps[[3]], MA3$n_ps[[3]])[2]
MA3$smd_re2[[3]] <- compute_gs(d, vd, n_ps, n_ps, MA3)$g[3]

# 4
MA3$smd_re2[[4]] <- compute_dgs(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[4,], denom = "pooled.sd")$g

#5 
MA3$mc_ps.3 <- (MA3$mc_ps.1 + MA3$mc_ps.2) / 2
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdc_ps.3 <- (MA3$sdc_ps.1 + MA3$sdc_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
MA3$smd_re2[[5]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA3[5,], denom = "pooled.sd")$g

#6
MA3$d <- p_to_d2(MA3$p_ps[[6]], MA3$nc_ps[[6]], MA3$nt_ps[[6]])[1]
MA3$vd <- p_to_d2(MA3$p_ps[[6]], MA3$nc_ps[[6]], MA3$nt_ps[[6]])[2]
MA3$smd_re2[[6]] <- compute_gs(d, vd, nt_ps, nc_ps, MA3)$g[6]

# make numeric
MA3 <- MA3 %>% mutate_at(vars(starts_with('smd')), as.numeric)

# relocate to facilitate inspection
MA3 <- MA3 %>% relocate(smd_re2,  .after = smd_re1)

3.4.2 Sampling variances

# 1
MA3$d <- p_to_d2(0.01, MA3$nt_ps[[1]], MA3$nc_ps[[1]])[1]
MA3$vd <- p_to_d2(0.01, MA3$nt_ps[[1]], MA3$nc_ps[[1]])[2]
MA3$v_re2[[1]] <- compute_gs(d, vd, nt_ps, nc_ps, MA3)$var.g[1]

# 2
MA3$v_re2[[2]] <- compute_dgs(nt_ps*2, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[2,], denom = "pooled.sd")$var.g

# 3
MA3$d <- p_to_d2(0.01, MA3$n_ps[[3]], MA3$n_ps[[3]])[1]
MA3$vd <- p_to_d2(0.01, MA3$n_ps[[3]], MA3$n_ps[[3]])[2]
MA3$v_re2[[3]] <- compute_gs(d, vd, n_ps, n_ps, MA3)$var.g[3]

# 4
MA3$v_re2[[4]] <- compute_dgs(nt_ps, mt_ps.1, sdt_ps.1, nc_ps, mc_ps.1, sdc_ps.1, data = MA3[4,], denom = "pooled.sd")$var.g

#5 
MA3$mc_ps.3 <- (MA3$mc_ps.1 + MA3$mc_ps.2) / 2
MA3$mt_ps.3 <- (MA3$mt_ps.1 + MA3$mt_ps.2) / 2
MA3$sdc_ps.3 <- (MA3$sdc_ps.1 + MA3$sdc_ps.2) / 2
MA3$sdt_ps.3 <- (MA3$sdt_ps.1 + MA3$sdt_ps.2) / 2
MA3$v_re2[[5]] <- compute_dgs(nt_ps, mt_ps.3, sdt_ps.3, nc_ps, mc_ps.3, sdc_ps.3, data = MA3[5,], denom = "pooled.sd")$var.g

#6
MA3$d <- p_to_d2(0.169, MA3$nc_ps[[6]], MA3$nt_ps[[6]])[1]
MA3$vd <- p_to_d2(0.169, MA3$nc_ps[[6]], MA3$nt_ps[[6]])[2]
MA3$v_re2[[6]] <- compute_gs(d, vd, nt_ps, nc_ps, MA3)$var.g[6]

# make numeric
MA3 <- MA3 %>% mutate_at(vars(starts_with('v_')), as.numeric)

3.5 Compute sampling variances and SEs based on the reported CIs

MA3$v_ma <- ((MA3$ciu_ma - MA3$smd_ma) / 1.96)^2
MA3$se_ma <- ((MA3$ciu_ma - MA3$smd_ma) / 1.96)

# compare with reproduced variances
MA3 <- MA3 %>% relocate(c(v_re1, v_re2, v_ma), .after = smd_ma)

3.6 Reproduce the pooled SMD

3.6.1 MA model 1

ma1 <- metagen(TE = smd_re1,
              seTE = sqrt(v_re1),
              studlab = id_comparison,
              data = MA3,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )



forest(ma1,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "g", "SE"))

3.6.2 MA model 2

ma2 <- metagen(TE = smd_re2,
              seTE = sqrt(v_re2),
              studlab = id_comparison,
              data = MA3,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA
              method.tau = "DL", # DerSimonian-Laird estimator
                 )


forest(ma2,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "g", "SE"))

3.6.3 MA model 3

ma3 <- metagen(TE = smd_ma,
              seTE = se_ma,
              studlab = as.character(id_comparison),
              data = MA3,
              sm = "SMD",
              fixed = FALSE,
              random = TRUE, # a random effects MA.
              method.tau = "DL", # DerSimonian-Laird estimator
                 )


forest(ma3,
           fontfamily = "Times",
           print.tau2 = TRUE,
           leftlabs = c("SMD No.", "g", "SE"))

4 Reproducibility summary plots

4.1 Summary stats

MA_all <- MA1 %>% 
  bind_rows(MA2) %>% 
  bind_rows(MA3) %>% 
  mutate(dif = smd_ma - smd_re1,
         abs_dif = abs(smd_ma - smd_re1))

saveRDS(MA_all, "data/preprocessed/MA_all.rds")

mean(MA_all$dif, na.rm = T)
## [1] 0.161
max(MA_all$dif, na.rm = T)
## [1] 2.99
min(MA_all$dif, na.rm = T)
## [1] -1.08
mean(MA_all$abs_dif, na.rm = T)
## [1] 0.372
max(MA_all$abs_dif, na.rm = T)
## [1] 2.99
min(MA_all$abs_dif, na.rm = T)
## [1] 0.000464

4.2 Scatter plot, faithfully reproduced SMDs

MA_all <- MA_all %>% 
  mutate(id_ma = as.character(id_ma))

ggplot(MA_all, aes(smd_ma, round(smd_re1, 2))) +
  geom_abline(slope = 1, linetype = "dotted") +
  geom_point(aes(color = id_ma, shape = id_ma), size = 2.25) +
  theme_classic(base_family = "Times", base_size = 12)+
  labs(
    x = "Reported primary ES",
    y = "Reproduced primary ES",
    color = "Meta-analysis ID",
    shape = "Meta-analysis ID"
  ) +
annotate("text", x = 0, y = 1.80, size = 4, label = "Undererstimates ES", family = "Times") +
annotate("text", x = 1, y = -.5, size = 4, label = "Overestimates ES", family = "Times") +
scale_colour_manual(values = c("#009e73", "#0072b2", "#e69f00")) + theme(aspect.ratio=1)

4.3 Scatter plot, all reproduced SMDs

ggplot(MA_all, aes(smd_ma, round(smd_re2, 2))) +
  geom_abline(slope = 1, linetype = "dotted") +
  geom_point(aes(color = id_ma, shape = id_ma), size = 2.25) +
  theme_classic(base_family = "Times", base_size = 12)+
  labs(
    x = "Reported primary SMD",
    y = "Faithfully & brute-force reproduced primary SMD",
    color = "Meta-analysis ID",
    shape = "Meta-analysis ID"
  ) +
annotate("text", x = 0, y = 1.80, size = 4, label = "Underestimates ES", family = "Times") +
annotate("text", x = 1, y = -.5, size = 4, label = "Overestimates ES", family = "Times") +
scale_colour_manual(values = c("#009e73", "#0072b2", "#e69f00")) + theme(aspect.ratio=1)