run_workflow() chains the data, preprocessing,
selection, modelling, evaluation, and result layers into one object. The
held-out validation and test sets are transformed with parameters
learned from the training set; they are never used to fit preprocessing,
selection, calibration, or thresholds.
The vignette uses the bundled cardiovascular example cohort. It is deliberately trimmed here so the vignette builds quickly; the full cohort has 2000 rows per data set.
library(uniml)
development <- read.csv(
system.file("extdata", "uniml-cvd-mortality.csv", package = "uniml")
)
new_admissions <- read.csv(
system.file("extdata", "uniml-cvd-mortality-new.csv", package = "uniml")
)
set.seed(123)
development <- development[1:800, ]
new_admissions <- new_admissions[1:400, ]
spec <- uniml_spec(
task_type = "binary",
outcome = "died",
positive = "yes",
id = "id"
)
spec <- set_preprocess(
spec,
impute = "median_mode",
scale = "zscore",
encode = "onehot"
)
spec <- set_feature_selection(
spec,
methods = c("univariate", "lasso"),
combine = "intersect"
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_evaluation(
spec,
ci = "bootstrap",
threshold_method = "youden",
calibration = "platt",
select_on = "validation"
)
# Kernel SHAP is the most expensive step of a workflow, so its two size knobs are
# part of the specification. They are small here to keep the vignette quick; a
# paper-quality explanation wants a few hundred rows on both sides.
spec <- set_explain(spec, n_explain = 20, bg_n = 20)
spec
#> <uniml_spec>
#> task_type: binary
#> outcome: died
#> positive: yes
#> leakage_policy: custom
#> in_cv: selection, balance, tuning
#> selection: univariate, lasso
#> verbose: 1, n_workers: 1The built-in learner keys are lr, rf,
xgb, nnet, knn,
cart, and nb, plus the optional
log_reg, svm, lda, and
qda. Any installed mlr3 classification learner can also be
passed directly, such as classif.log_reg,
classif.svm, or classif.lda. Learners from
packages that are not on CRAN (for example classif.lightgbm
from mlr3extralearners) work the same way once that package
is installed and loaded; external learners use their mlr3 defaults
unless a custom search space is registered.
make_split() returns a uniml_split whose
elements are uniml_data objects. Pass the
$data component to run_workflow(). The
independent new-admission cohort stays outside the development
split.
split <- make_split(
as_uniml_data(development, spec),
prop = c(0.8, 0.2),
stratify = TRUE,
seed = 123
)
data.frame(
set = c("train", "validation"),
n = c(nrow(split$train$data), nrow(split$validation$data)),
events = c(
sum(split$train$data$died == "yes"),
sum(split$validation$data$died == "yes")
)
)
#> set n events
#> 1 train 641 45
#> 2 validation 159 11The first 200 new admissions play the role of an independent test set, the next 200 the role of an external cohort. Both are reporting-only.
result <- run_workflow(
spec,
train = split$train$data,
validation = split$validation$data,
test = new_admissions[1:200, ],
external = new_admissions[201:400, ],
ci_R = 25
)
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: dor.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_slope,
#> calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: f1, f2,
#> calib_slope, calib_intercept, calib_large.
#> Warning: Fewer than 80% of bootstrap resamples were effective for: calib_large.
result
#> <uniml_result>
#> data_info (list, 7 elements)
#> preprocess (list, 4 elements)
#> selection (list, 8 elements)
#> tuning (list, 2 elements)
#> cv (list, 6 elements)
#> metrics (16 x 24)
#> ci (336 x 10)
#> calibration (list, 4 elements)
#> tests (list, 3 elements)
#> dca (1188 x 6)
#> ranking (2 x 6)
#> shap (list, 12 elements)
#> plots (list, 4 elements)
#> models (list, 2 elements)
#> meta (list, 13 elements)result$metrics is a wide table with one row per model,
set, and probability track (raw or calibrated):
metrics <- result$metrics
metrics[, c("model", "set", "probability", "auc", "brier", "accuracy")]
#> model set probability auc brier accuracy
#> 1 lr train raw 0.9035421 0.05103773 0.7722309
#> 2 lr train calibrated 0.9035421 0.04995033 0.7722309
#> 3 lr validation raw 0.9066339 0.05000148 0.6918239
#> 4 lr validation calibrated 0.9066339 0.04755764 0.6918239
#> 5 lr test raw 0.9075521 0.03376129 0.7650000
#> 6 lr test calibrated 0.9075521 0.03610441 0.7650000
#> 7 lr external raw 0.8864839 0.04105926 0.7700000
#> 8 lr external calibrated 0.8864839 0.04096392 0.7700000
#> 9 cart train raw 0.6836689 0.04752045 0.9438378
#> 10 cart train calibrated 0.6836689 0.05060635 0.9438378
#> 11 cart validation raw 0.5816953 0.05935709 0.9245283
#> 12 cart validation calibrated 0.5816953 0.05848193 0.9245283
#> 13 cart test raw 0.5530599 0.04209931 0.9450000
#> 14 cart test calibrated 0.5530599 0.04046960 0.9450000
#> 15 cart external raw 0.6269841 0.04804633 0.9400000
#> 16 cart external calibrated 0.6269841 0.04827002 0.9400000Bootstrap confidence intervals are stored separately so that they can be exported on their own:
head(result$ci[result$ci$set == "validation", ], 4)
#> model set probability metric mean se lower
#> 43 lr validation raw auc 0.9170925 0.03262267 0.85815355
#> 44 lr validation raw pr_auc 0.5428291 0.13539597 0.24476019
#> 45 lr validation raw brier 0.0480432 0.01246494 0.03152822
#> 46 lr validation raw accuracy 0.6948428 0.03358161 0.62515723
#> upper n_effective formatted
#> 43 0.97748263 25 0.917 (0.858-0.977)
#> 44 0.73160017 25 0.543 (0.245-0.732)
#> 45 0.07209281 25 0.048 (0.032-0.072)
#> 46 0.75094340 25 0.695 (0.625-0.751)The ranking selects on the comparison set (validation by default) and never on the test or external cohorts:
result$ranking[, c("model", "score", "rank", "metric", "set")]
#> model score rank metric set
#> 1 lr 0.7264577 1 composite validation
#> 2 cart 0.4482745 2 composite validationCross-validation output is available per model and metric:
All plotting helpers accept palette, theme,
title, subtitle, caption, and
set, so figures can be styled for a manuscript without
touching the result.
plot_calibration(result, set = "validation")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?The three held-out sets answer different questions, so they are reported separately and never pooled. Model selection uses the validation set only; the test set and the external cohort are reporting-only.
role_table <- subset(
result$metrics,
set %in% c("validation", "test", "external"),
select = c("model", "set", "probability", "auc", "brier", "sensitivity",
"specificity")
)
role_table[role_table$probability == "calibrated", ]
#> model set probability auc brier sensitivity specificity
#> 4 lr validation calibrated 0.9066339 0.04755764 0.9090909 0.6756757
#> 6 lr test calibrated 0.9075521 0.03610441 1.0000000 0.7552083
#> 8 lr external calibrated 0.8864839 0.04096392 0.8181818 0.7671958
#> 12 cart validation calibrated 0.5816953 0.05848193 0.1818182 0.9797297
#> 14 cart test calibrated 0.5530599 0.04046960 0.1250000 0.9791667
#> 16 cart external calibrated 0.6269841 0.04827002 0.2727273 0.9788360Discrimination on the external cohort, which took no part in training, feature selection, or calibration:
Comparing the calibration curves of the validation and external cohorts shows whether the probabilities travel; a model that only looks good on the development split is usually overfitted to it.
plot_calibration(result, set = "external")
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?
#> `geom_line()`: Each group consists of only one observation.
#> ℹ Do you need to adjust the group aesthetic?The classification threshold is derived from out-of-fold training predictions, so it is not tuned on the validation data:
as.data.frame(result$meta$thresholds)
#> lr.raw lr.calibrated cart.raw cart.calibrated
#> 1 0.06938979 0.04994531 0.05263158 0.07681389When two or more models are compared, NRI and IDI are computed against the reference model (the best training AUC by default):
SHAP explanations use kernelshap; the number of
explained rows and the background sample are controlled by
n_explain and bg_n:
explained <- explain_shap(result, model = "best", n_explain = 30, bg_n = 20)
plot_shap_importance(explained)predict_uniml() reuses the stored preprocessing,
selected feature mapping, calibration, threshold, and fitted learner. It
does not refit anything.
predicted <- predict_uniml(
result,
newdata = new_admissions[1:5, ],
model = "best"
)
predicted[, c("model", "prob", "calibrated_prob", "probability", "predicted")]
#> model prob calibrated_prob probability predicted
#> 1 lr 0.01132240 0.002474690 calibrated 0
#> 2 lr 0.02432465 0.008738865 calibrated 0
#> 3 lr 0.03504350 0.016014444 calibrated 0
#> 4 lr 0.04703144 0.026130130 calibrated 0
#> 5 lr 0.16267780 0.200254276 calibrated 1The newdata argument may omit the outcome column. In
that case the returned data frame contains predicted probabilities and
classes without a truth column.
tripod_ai_check() audits the result against the
TRIPOD+AI domains and states where each piece of information lives. It
never claims compliance: items such as study design, subgroup fairness,
and code availability must be written by the author.
checklist <- tripod_ai_check(result)
table(checklist$status)
#>
#> reported partial not_reported not_supported
#> 12 1 1 4
subset(checklist, status != "reported", select = c("domain", "status"))
#> domain status
#> 1 Data source and setting not_supported
#> 2 Participants not_supported
#> 12 Calibration not_reported
#> 16 Fairness and subgroups not_supported
#> 17 Reproducibility partial
#> 18 Code and data availability not_supportedexport_results() writes the tables, figures, and fitted
models, plus the reporting checklist, to a directory that can be used
outside R:
out <- tempfile("uniml-vignette-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
#> [1] "export_manifest.csv" "result.rds"
#> [3] "var_info.csv" "scaling_params.csv"
#> [5] "encoder_map.csv" "selected_features.csv"
#> [7] "by_method_detail.csv" "best_hyperparameters.csv"
#> [9] "cv_summary.csv" "cv_detailed.csv"
#> [11] "metrics_train.csv" "metrics_validation.csv"
#> [13] "metrics_test.csv" "metrics_external.csv"
#> [15] "metrics_ci.csv" "before_after_calibration.csv"
#> [17] "thresholds.csv" "predictions.csv"
#> [19] "leaderboard.csv" "overall_rank.csv"
#> [21] "params.csv" "ece_brier.csv"
#> [23] "delong.csv" "nri_idi.csv"
#> [25] "dca_curve.csv" "failures.csv"
#> [27] "tripod_ai.csv" "importance.csv"
#> [29] "values_matrix_std.csv" "values_matrix_raw.csv"
#> [31] "feature_values_std.csv" "feature_values_raw.csv"
#> [33] "final_lr.rds" "final_cart.rds"The default is a transparent custom policy. It can be changed explicitly:
spec_in <- set_leakage_policy(
spec,
mode = "strict_in",
in_cv = c("selection", "balance", "tuning")
)
spec_in$leakage_policy
#> [1] "strict_in"
spec_in$in_cv
#> [1] "selection" "balance" "tuning"strict_in repeats configured steps inside each
cross-validation fold. strict_out keeps those steps outside
the fold loop. The validation and test sets remain held out in both
cases.
See system.file("REFERENCES.md", package = "uniml") for
the methods and reporting guidelines that this workflow implements.