Test and effect size details

Introduction

This article describes the data that {statsExpressions} functions expect, and summarizes, for each function, the statistical tests carried out, the effect sizes returned, and the underlying functions used to compute them.

Abbreviations used: CI = Confidence Interval

Data requirements

The one-sample, group-comparison, correlation, and centrality functions expect long (tidy) data — one row per observation. Contingency-table functions also accept a counts column for aggregated frequencies; meta_analysis() instead requires estimate and std.error columns, with one row per study. A few additional requirements are worth noting:

Summary of functionality

Function Analysis Parametric Non-parametric Robust Bayesian
one_sample_test() one-sample test ✅ ✅ ✅ ✅
two_sample_test() two-sample test ✅ ✅ ✅ ✅
oneway_anova() one-way ANOVA ✅ ✅ ✅ ✅
corr_test() correlation ✅ ✅ ✅ ✅
contingency_table() one- or two-way contingency table ✅ ✅ ❌ ✅
meta_analysis() random-effects meta-analysis ✅ ❌ ✅ ✅
pairwise_comparisons() pairwise group comparisons ✅ ✅ ✅ ✅
centrality_description() centrality and distribution ✅ ✅ ✅ ✅

❌ means that no dedicated analysis of that type exists. For contingency tables, type = "nonparametric" and type = "robust" both use the same frequentist tests as type = "parametric".

pairwise_contingency_table() provides pairwise Fisher’s exact tests with Cramer’s V effect sizes. It does not have a type argument.

Summary of tests and effect sizes

Here is a go-to summary of the statistical tests carried out and the effect sizes returned by each function. This should be useful if one needs to find out more information about how an argument is resolved in the underlying package or if one wishes to browse the source code. So, for example, if you want to know more about how the one-way (between-subjects) ANOVA is carried out, you can run ?stats::oneway.test in your R console.

centrality_description()

Type Measure Function used
Parametric mean datawizard::describe_distribution()
Non-parametric median datawizard::describe_distribution()
Robust trimmed mean datawizard::describe_distribution()
Bayesian MAP datawizard::describe_distribution()

oneway_anova()

Between-subjects

Hypothesis testing

Type No. of groups Test Function used
Parametric > 2 Fisher’s or Welch’s one-way ANOVA stats::oneway.test()
Non-parametric > 2 Kruskal-Wallis one-way ANOVA stats::kruskal.test()
Robust > 2 Heteroscedastic one-way ANOVA for trimmed means WRS2::t1way()
Bayesian > 2 Fisher’s ANOVA BayesFactor::anovaBF()

Effect size estimation

Type No. of groups Effect size CI available? Function used
Parametric > 2 partial eta-squared, partial omega-squared Yes effectsize::omega_squared(), effectsize::eta_squared()
Non-parametric > 2 rank epsilon squared Yes effectsize::rank_epsilon_squared()
Robust > 2 Explanatory measure of effect size Yes WRS2::t1way()
Bayesian > 2 Bayesian R-squared Yes performance::r2_bayes()

Within-subjects

Paired data: See complete_subjects() for subject matching, duplicate observations, and removal of incomplete subjects.

Hypothesis testing

Type No. of groups Test Function used
Parametric > 2 One-way repeated measures ANOVA afex::aov_ez()
Non-parametric > 2 Friedman rank sum test stats::friedman.test()
Robust > 2 Heteroscedastic one-way repeated measures ANOVA for trimmed means WRS2::rmanova()
Bayesian > 2 One-way repeated measures ANOVA BayesFactor::anovaBF()

Effect size estimation

Type No. of groups Effect size CI available? Function used
Parametric > 2 partial eta-squared, partial omega-squared Yes effectsize::omega_squared(), effectsize::eta_squared()
Non-parametric > 2 Kendall’s coefficient of concordance Yes effectsize::kendalls_w()
Robust > 2 Algina-Keselman-Penfield robust standardized difference average Yes WRS2::wmcpAKP()
Bayesian > 2 Bayesian R-squared Yes performance::r2_bayes()

two_sample_test()

Between-subjects

Hypothesis testing

Type No. of groups Test Function used
Parametric 2 Student’s or Welch’s t-test stats::t.test()
Non-parametric 2 Mann-Whitney U test stats::wilcox.test()
Robust 2 Yuen’s test for trimmed means WRS2::yuen()
Bayesian 2 Student’s t-test BayesFactor::ttestBF()

Effect size estimation

Type No. of groups Effect size CI available? Function used
Parametric 2 Cohen’s d, Hedges’ g Yes effectsize::cohens_d(), effectsize::hedges_g()
Non-parametric 2 r (rank-biserial correlation) Yes effectsize::rank_biserial()
Robust 2 Algina-Keselman-Penfield robust standardized difference Yes WRS2::akp.effect()
Bayesian 2 difference Yes bayestestR::describe_posterior()

Within-subjects

Paired data: See complete_subjects() for subject matching, duplicate observations, and removal of incomplete subjects.

Hypothesis testing

Type No. of groups Test Function used
Parametric 2 Student’s t-test stats::t.test()
Non-parametric 2 Wilcoxon signed-rank test stats::wilcox.test()
Robust 2 Yuen’s test on trimmed means for dependent samples WRS2::yuend()
Bayesian 2 Student’s t-test BayesFactor::ttestBF()

Effect size estimation

Type No. of groups Effect size CI available? Function used
Parametric 2 Cohen’s d, Hedges’ g Yes effectsize::cohens_d(), effectsize::hedges_g()
Non-parametric 2 r (rank-biserial correlation) Yes effectsize::rank_biserial()
Robust 2 Algina-Keselman-Penfield robust standardized difference Yes WRS2::wmcpAKP()
Bayesian 2 difference Yes bayestestR::describe_posterior()

one_sample_test()

Hypothesis testing

Type Test Function used
Parametric One-sample Student’s t-test stats::t.test()
Non-parametric One-sample Wilcoxon test stats::wilcox.test()
Robust Bootstrap-t method for one-sample test WRS2::trimcibt()
Bayesian One-sample Student’s t-test BayesFactor::ttestBF()

Effect size estimation

Type Effect size CI available? Function used
Parametric Cohen’s d, Hedges’ g Yes effectsize::cohens_d(), effectsize::hedges_g()
Non-parametric r (rank-biserial correlation) Yes effectsize::rank_biserial()
Robust trimmed mean Yes WRS2::trimcibt()
Bayesian difference Yes bayestestR::describe_posterior()

corr_test()

Hypothesis testing and effect size estimation

Type Test CI available? Function used
Parametric Pearson’s correlation coefficient Yes correlation::correlation()
Non-parametric Spearman’s rank correlation coefficient Yes correlation::correlation()
Robust Winsorized Pearson’s correlation coefficient Yes correlation::correlation()
Bayesian Bayesian Pearson’s correlation coefficient Yes correlation::correlation()

contingency_table()

There are no dedicated non-parametric or robust contingency table analyses. type = "nonparametric" and type = "robust" are accepted, but run the same frequentist tests as type = "parametric" (the “Parametric/Non-parametric” rows below).

Two-way table

Hypothesis testing

Type Design Test Function used
Parametric/Non-parametric Unpaired Pearson’s chi-squared test stats::chisq.test()
Bayesian Unpaired Bayesian Pearson’s chi-squared test BayesFactor::contingencyTableBF()
Parametric/Non-parametric Paired McNemar’s chi-squared test stats::mcnemar.test()
Bayesian Paired No No

Effect size estimation

Type Design Effect size CI available? Function used
Parametric/Non-parametric Unpaired Cramer’s V Yes effectsize::cramers_v()
Bayesian Unpaired Cramer’s V Yes effectsize::cramers_v()
Parametric/Non-parametric Paired Cohen’s g Yes effectsize::cohens_g()
Bayesian Paired No No No

Paired Bayesian analysis is not supported: for a two-way table with type = "bayes", the paired argument is ignored and the unpaired Bayesian test is run instead.

One-way table

Hypothesis testing

Type Test Function used
Parametric/Non-parametric Goodness of fit chi-squared test stats::chisq.test()
Bayesian Bayesian Goodness of fit chi-squared test (custom)

Effect size estimation

Type Effect size CI available? Function used
Parametric/Non-parametric Pearson’s C Yes effectsize::pearsons_c()
Bayesian No No No

pairwise_comparisons()

Between-subjects

Hypothesis testing

Type Equal variance? Test p-value adjustment? Function used
Parametric No Games-Howell test Yes PMCMRplus::gamesHowellTest()
Parametric Yes Student’s t-test Yes stats::pairwise.t.test()
Non-parametric No Dunn test Yes PMCMRplus::kwAllPairsDunnTest()
Robust No Yuen’s trimmed means test Yes WRS2::lincon()
Bayesian NA Student’s t-test NA BayesFactor::ttestBF()

Effect size estimation

Not supported.

Within-subjects

Paired data: See complete_subjects() for subject matching, duplicate observations, and removal of incomplete subjects.

Hypothesis testing

Type Test p-value adjustment? Function used
Parametric Student’s t-test Yes stats::pairwise.t.test()
Non-parametric Durbin-Conover test Yes PMCMRplus::durbinAllPairsTest()
Robust Yuen’s trimmed means test Yes WRS2::rmmcp()
Bayesian Student’s t-test NA BayesFactor::ttestBF()

Effect size estimation

Not supported.

pairwise_contingency_table()

Hypothesis testing

Test p-value adjustment? Function used
Fisher’s exact test Yes stats::fisher.test()

Effect size estimation

Effect size CI available? Function used
Cramer’s V Yes effectsize::cramers_v()

meta_analysis()

Hypothesis testing and effect size estimation

Type Test Effect size CI available? Function used
Parametric Meta-analysis via random-effects models beta Yes metafor::rma()
Robust Meta-analysis via robust random-effects models beta Yes metaplus::metaplus()
Bayesian Meta-analysis via Bayesian random-effects models beta Yes metaBMA::meta_random()

Effect size interpretation

See {effectsize}’s interpretation functions to check different rules/conventions to interpret effect sizes:

https://easystats.github.io/effectsize/reference/index.html#section-interpretation

References

Suggestions

If you find any bugs or have any suggestions/remarks, please file an issue on GitHub: https://github.com/IndrajeetPatil/statsExpressions/issues

Citation

To cite package 'statsExpressions' in publications use:

  Patil, I., (2021). statsExpressions: R Package for Tidy Dataframes
  and Expressions with Statistical Details. Journal of Open Source
  Software, 6(61), 3236, https://doi.org/10.21105/joss.03236

A BibTeX entry for LaTeX users is

  @Article{,
    doi = {10.21105/joss.03236},
    year = {2021},
    publisher = {{The Open Journal}},
    volume = {6},
    number = {61},
    pages = {3236},
    author = {Indrajeet Patil},
    title = {{statsExpressions: {R} Package for Tidy Dataframes and Expressions with Statistical Details}},
    journal = {{Journal of Open Source Software}},
  }