Every other example in this package uses a cohort that
data-raw/ simulates. That makes the code reproducible, but
it says nothing about how the workflow behaves on real data, where
probabilities are lumpy, groups can be empty, and a hundred and thirteen
patients is all you have.
This vignette runs the same workflow on a published clinical data set
that ships with pROC, so the numbers below are real and you
can check them against the source paper.
pROC::aSAH contains 113 patients with aneurysmal
subarachnoid haemorrhage. The outcome is the dichotomised Glasgow
Outcome Scale at six months, and the candidate predictors are two blood
markers (s100b, ndka) plus age, sex, and the
World Federation of Neurosurgical Societies grade.
asah <- pROC::aSAH
str(asah[, c("outcome", "gos6", "gender", "age", "wfns", "s100b", "ndka")])
#> 'data.frame': 113 obs. of 7 variables:
#> $ outcome: Factor w/ 2 levels "Good","Poor": 1 1 1 1 2 2 1 2 1 1 ...
#> $ gos6 : Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 5 5 5 5 1 1 4 1 5 4 ...
#> $ gender : Factor w/ 2 levels "Male","Female": 2 2 2 2 2 1 1 1 2 2 ...
#> $ age : int 42 37 42 27 42 48 57 41 49 75 ...
#> $ wfns : Ord.factor w/ 5 levels "1"<"2"<"3"<"4"<..: 1 1 1 1 3 2 5 4 1 2 ...
#> $ s100b : num 0.13 0.14 0.1 0.04 0.13 0.1 0.47 0.16 0.18 0.1 ...
#> $ ndka : num 3.01 8.54 8.09 10.42 17.4 ...
table(asah$outcome)
#>
#> Good Poor
#> 72 41Note gos6: it is the five-level version of the outcome,
and outcome is exactly gos6 collapsed to two
groups.
Leaving gos6 in the predictor set would let the model
read the answer off the answer sheet. It goes in exclude,
and the specification refuses to use it:
library(uniml)
spec <- uniml_spec(
task_type = "binary",
outcome = "outcome",
positive = "Poor",
exclude = "gos6"
)
spec <- set_preprocess(
spec,
impute = "median_mode",
scale = "zscore",
encode = "onehot"
)
spec <- set_feature_selection(
spec,
methods = "univariate",
p_threshold = 0.2
)
spec <- set_learners(spec, c("lr", "cart"), tune = FALSE)
spec <- set_tuning(spec, folds = 5, repeats = 5, evals = 4, metric = "auc", seed = 123)
spec <- set_evaluation(
spec,
ci = "bootstrap",
calibration = "platt",
threshold_method = "youden",
select_on = "validation"
)
spec <- set_leakage_policy(spec, mode = "strict_in")
spec <- set_explain(spec, model = "none", plots = character())
spec
#> <uniml_spec>
#> task_type: binary
#> outcome: outcome
#> positive: Poor
#> leakage_policy: strict_in
#> in_cv: imputation, scaling, encoding, selection, balance, tuning
#> selection: univariate
#> verbose: 1, n_workers: 1strict_in re-runs the univariate selection inside every
fold, so the cross-validated numbers do not profit from having seen the
whole training set. With five predictors that costs almost nothing.
Seventy per cent of the cohort develops the models, thirty per cent is held out for the whole workflow.
split <- make_split(
as_uniml_data(asah, spec),
prop = c(0.7, 0.3),
stratify = TRUE,
seed = 123
)
data.frame(
set = c("train", "validation"),
n = c(nrow(split$train$data), nrow(split$validation$data)),
poor_outcome = c(
sum(split$train$data$outcome == "Poor"),
sum(split$validation$data$outcome == "Poor")
)
)
#> set n poor_outcome
#> 1 train 80 29
#> 2 validation 33 12result <- suppressWarnings(run_workflow(
spec,
train = split$train$data,
validation = split$validation$data,
ci_R = 40
))
result
#> <uniml_result>
#> data_info (list, 7 elements)
#> preprocess (list, 4 elements)
#> selection (list, 8 elements)
#> tuning (list, 2 elements)
#> cv (list, 6 elements)
#> metrics (8 x 24)
#> ci (168 x 10)
#> calibration (list, 4 elements)
#> tests (list, 3 elements)
#> dca (1188 x 6)
#> ranking (2 x 6)
#> models (list, 2 elements)
#> meta (list, 13 elements)Nothing was dropped, downgraded, or silently skipped:
The chunks above suppress warnings. The ones that fired are expected
on data this small: bootstrap resamples in which the positive likelihood
ratio and the diagnostic odds ratio are infinite, and a Platt fit on
tree probabilities that are exactly 0 or 1. Neither is an error, and
both are visible in the output below — the affected intervals come back
as NA rather than as a number invented by the package.
Repeated cross-validation on the 80 development patients is the most stable estimate available here.
cv_auc <- subset(result$cv$summary, metric == "auc")
cv_auc[, c("model", "mean", "sd", "lower", "upper", "n")]
#> model mean sd lower upper n
#> 1 lr 0.7857858 0.09543635 0.7513352 0.8202363 5
#> 22 cart 0.7287635 0.14001758 0.7073763 0.7501507 5n = 5 is the number of repeats: each repeat averages
five folds, and the interval summarises spread across those five repeat
means. It is narrower than the sampling uncertainty of the AUC itself,
so do not read it as a confidence interval for new patients — compare it
with the bootstrap interval on the validation patients further down,
which is much wider.
The held-out set has 33 patients and 12 events.
validation <- subset(
result$metrics,
set == "validation",
select = c("model", "probability", "auc", "brier", "sensitivity",
"specificity", "ppv", "npv")
)
validation
#> model probability auc brier sensitivity specificity ppv
#> 3 lr raw 0.8571429 0.1689515 0.4166667 1.0000000 1.0000
#> 4 lr calibrated 0.8571429 0.1502564 0.4166667 1.0000000 1.0000
#> 7 cart raw 0.8115079 0.1776383 0.7500000 0.6666667 0.5625
#> 8 cart calibrated 0.8115079 0.1639931 0.7500000 0.6666667 0.5625
#> npv
#> 3 0.7500000
#> 4 0.7500000
#> 7 0.8235294
#> 8 0.8235294Bootstrap intervals around the same numbers:
ci <- subset(
result$ci,
set == "validation" & metric %in% c("auc", "brier"),
select = c("model", "probability", "metric", "mean", "lower", "upper")
)
ci
#> model probability metric mean lower upper
#> 43 lr raw auc 0.8430825 0.67541322 0.9635862
#> 45 lr raw brier 0.1718083 0.11203441 0.2270106
#> 64 lr calibrated auc 0.8430825 0.67541322 0.9635862
#> 66 lr calibrated brier 0.1552310 0.09028117 0.2167864
#> 127 cart raw auc 0.8062441 0.61152597 0.9251188
#> 129 cart raw brier 0.1753687 0.08648452 0.2435695
#> 148 cart calibrated auc 0.8062441 0.61152597 0.9251188
#> 150 cart calibrated brier 0.1614905 0.08523515 0.2240351The logistic model calibrates better than the tree: Platt scaling
cuts its validation Brier score by about 0.019, and cart’s
by about 0.014. Reads of a single 33-patient split should be treated as
illustrative; the cross-validated table above is what to quote.
Two of the ratios cannot be given an interval on this data, and the package says so rather than inventing bounds:
subset(
result$ci,
set == "validation" & metric %in% c("plr", "dor"),
select = c("model", "metric", "mean", "lower", "upper", "n_effective")
)
#> model metric mean lower upper n_effective
#> 54 lr plr NA NA NA 0
#> 56 lr dor NA NA NA 0
#> 75 lr plr NA NA NA 0
#> 77 lr dor NA NA NA 0
#> 138 cart plr 2.808466 1.232143 5.13000 40
#> 140 cart dor 11.380938 1.437500 36.45833 36
#> 159 cart plr 2.808466 1.232143 5.13000 40
#> 161 cart dor 11.380938 1.437500 36.45833 36lr calls every validation patient at low risk, so its
specificity is exactly 1 and the positive likelihood ratio is infinite
in all 200 resamples. The interval comes back as NA with
n_effective = 0. cart keeps a finite ratio and
therefore a finite interval.
rank_models() scores each model on the comparison set
(validation by default) using the calibrated probabilities, averaging
AUC, sensitivity, specificity, F2, balanced accuracy, and Youden’s
index.
result$ranking[, c("model", "score", "rank", "metric", "set")]
#> model score rank metric set
#> 1 cart 0.6760499 1 composite validation
#> 2 lr 0.6450846 2 composite validationcart comes first. It has the lower AUC but the much
higher sensitivity at the Youden threshold, and the composite weights
the two equally. That is a choice, not a law: pass
best_model_metric = "auc" to rank_models() if
discrimination is all you care about, and pick deliberately rather than
by default.
The classification threshold comes from out-of-fold training predictions, never from the validation patients.
result$tests$delong$summary
#> model auc
#> 1 lr 0.8571429
#> 2 cart 0.8115079
result$tests$delong$formatted
#> lr cart
#> lr "-" "0.3616"
#> cart "0.3616" "-"The DeLong test compares AUCs on the same patients and accounts for the correlation between the two prediction vectors. With 12 events it has very little power, so a non-significant p value means “not shown to differ”, not “equivalent”.
for (track in c("raw", "calibrated")) {
for (model in c("lr", "cart")) {
item <- result$tests$hl[[track]][[model]]
cat(sprintf(
"%-10s %-5s statistic = %s, df = %s, p = %s (%s)\n",
track, model,
format(round(item$statistic, 3)), format(item$df),
format(round(item$p_value, 3)), item$method
))
}
}
#> raw lr statistic = 4.907, df = 3, p = 0.179 (ResourceSelection)
#> raw cart statistic = NA, df = 2, p = NA (not_estimable)
#> calibrated lr statistic = 2.937, df = 3, p = 0.401 (ResourceSelection)
#> calibrated cart statistic = 0.924, df = 2, p = 0.63 (ResourceSelection)cart reports not_estimable on the raw
track: the tree puts a group of patients at a probability of exactly 1,
which leaves the expected non-event count at zero and the
Hosmer-Lemeshow statistic undefined. The package says so with a message
instead of returning NaN; the fix in practice is fewer
groups or more patients. After Platt scaling the same model has
spread-out probabilities and the statistic becomes estimable, which is a
fair summary of what calibration does.
subset(
result$tests$nri_idi,
set == "validation",
select = c("model", "reference", "nri", "nri_events", "nri_nonevents", "idi")
)
#> model reference nri nri_events nri_nonevents idi
#> 1 lr cart -0.7142857 0 -0.7142857 -0.1877569
#> 2 cart cart 0.0000000 0 0.0000000 0.0000000NRI and IDI are computed against the reference model (the best
training AUC, cart here). The result is worth reading
carefully: lr reclassifies none of the events and 71% of
the non-events downward, so its net reclassification is negative even
though its AUC on the same patients is higher. With twelve events,
neither number is evidence of anything — but the disagreement is exactly
why AUC alone is a poor basis for choosing a risk model.
checklist <- tripod_ai_check(result)
table(checklist$status)
#>
#> reported partial not_reported not_supported
#> 10 1 3 4
subset(checklist, status != "reported", select = c("domain", "status"))
#> domain status
#> 1 Data source and setting not_supported
#> 2 Participants not_supported
#> 10 External validation not_reported
#> 12 Calibration not_reported
#> 15 Explainability not_reported
#> 16 Fairness and subgroups not_supported
#> 17 Reproducibility partial
#> 18 Code and data availability not_supportedout <- tempfile("uniml-asah-")
exported <- export_results(result, dir = out, include_plots = FALSE)
basename(exported$files)
#> [1] "export_manifest.csv" "result.rds"
#> [3] "var_info.csv" "scaling_params.csv"
#> [5] "encoder_map.csv" "selected_features.csv"
#> [7] "by_method_detail.csv" "best_hyperparameters.csv"
#> [9] "cv_summary.csv" "cv_detailed.csv"
#> [11] "metrics_train.csv" "metrics_validation.csv"
#> [13] "metrics_test.csv" "metrics_ci.csv"
#> [15] "before_after_calibration.csv" "thresholds.csv"
#> [17] "predictions.csv" "leaderboard.csv"
#> [19] "overall_rank.csv" "params.csv"
#> [21] "ece_brier.csv" "delong.csv"
#> [23] "nri_idi.csv" "dca_curve.csv"
#> [25] "failures.csv" "tripod_ai.csv"
#> [27] "final_lr.rds" "final_cart.rds"It shows that the workflow runs end to end on real, complete clinical
data, that the leakage guard is not hypothetical (gos6
would have made every model look brilliant), and that the reporting
layer degrades honestly when the data is too small for a statistic to be
defined.
It does not show a validated model. A hundred and thirteen patients cannot support one, and the package is not a substitute for a prospective validation study. If you report from a cohort this size, report the cross-validated estimates with their intervals, say how wide they are, and do not select between models on thirty-three patients.
Sabbagh C, et al. (2005). Protein S100B and neuron-specific enolase
in subarachnoid haemorrhage. The aSAH data set is
distributed with the pROC package and used here under its
licence.
See system.file("REFERENCES.md", package = "uniml") for
the methods that each layer implements.