Getting started with datagrade

One call

dg_assess() takes a data frame or a file path and returns a dg_report object. Nothing is written to the global environment, and the data frame you pass in is never modified.

report <- dg_assess(iris, verbose = FALSE)
report
#> 
#> ── Data quality report ─────────────────────────────────────────────────────────
#> Source: in-memory data frame
#> Size: 150 rows x 5 columns
#> Elapsed: 0.03s
#> 
#> ── ISO/IEC 25012 characteristics ──
#> 
#> • accuracy: not applicable (0/3 measures)
#> • completeness: 1.000 (100%) - good (3/4 measures)
#> • consistency: 0.625 (62.5%) - serious (2/4 measures)
#> • credibility: not applicable (0/2 measures)
#> • currentness: not applicable (0/2 measures)
#> 
#> Overall: 0.812 (81.2%) - warning
#> 
#> Not ISO: plausibility 0.998 (99.8%) - outliers against a derived interval,
#> excluded from the overall score.
#> 
#> ℹ 10 of 15 measures need a declared expectation. See `report$measures` and `dg_spec()`.
#> 
#> ── Findings ──
#> 
#> • Missing values: 0 across 0 columns
#> • Duplicate rows: 1 (0.67%)
#> • Outliers (z_score): 1 of 600 numeric values
#> • No identifier column detected
#> • Redundant pairs above |r| > 0.5: 3 (strongest Petal.Length~Petal.Width at
#> 0.96)
#> • Suggested to drop: Petal.Length and Petal.Width
#> • VIF above 5: Sepal.Length, Petal.Length, and Petal.Width
#> • Non-normal columns (Shapiro-Wilk p < 0.05): Sepal.Length, Petal.Length, and
#> Petal.Width
#> 
#> ℹ `summary()` for tables, `dg_plots()` for figures.

The object, not the printout

The reference implementation this package grew out of printed everything with cat() and returned NULL, so nothing could be used programmatically. Here the printout is a side effect and the object is the product.

report$scores
#>     accuracy completeness  consistency  credibility  currentness 
#>           NA        1.000        0.625           NA           NA
report$measures[, c("code", "characteristic", "value", "a", "b")]
#>             code characteristic value   a   b
#> 5      EXAC_SINT       accuracy    NA  NA  NA
#> 6     EXAC_SEMAN       accuracy    NA  NA  NA
#> 7       RAN_EXAC       accuracy    NA  NA  NA
#> 1       COMP_REG   completeness  1.00 150 150
#> 2   COMP_VAL_ESP   completeness  1.00 750 750
#> 3      COMP_FICH   completeness    NA  NA  NA
#> 4  FAL_COMP_FICH   completeness  1.00 150 150
#> 8        INT_REF    consistency    NA  NA  NA
#> 9      RIES_INCO    consistency  0.25   1   4
#> 10    CONS_SEMAN    consistency    NA  NA  NA
#> 11     CONS_FORM    consistency  1.00 150 150
#> 12  CRED_VAL_DAT    credibility    NA  NA  NA
#> 13     CRED_FUEN    credibility    NA  NA  NA
#> 14      FREC_ACT    currentness    NA  NA  NA
#> 15      CONV_ACT    currentness    NA  NA  NA
report$outliers$by_column
#>         column n_outliers n_tested        rate
#> 1 Sepal.Length          0      150 0.000000000
#> 2  Sepal.Width          1      150 0.006666667
#> 3 Petal.Length          0      150 0.000000000
#> 4  Petal.Width          0      150 0.000000000
report$redundancy$pairs
#>       column_1     column_2 correlation
#> 1 Petal.Length  Petal.Width   0.9628654
#> 2 Sepal.Length Petal.Length   0.8717538
#> 3 Sepal.Length  Petal.Width   0.8179411

Each measure is a ratio A / B: A counts the data items satisfying the property, B the items for which the property is defined at all. A measure with b = NA is not a failing measure – it is one whose denominator could not be counted, because nobody has said what the data is supposed to look like yet.

Declaring what the data should look like

Six of the fifteen properties are computable from the table alone. The other nine compare the data against an expectation, and an expectation is declared, not discovered. dg_spec() is where it goes; every argument is optional and each one switches on the measures that depend on it.

spec <- dg_spec(
  required = c("Sepal.Length", "Species"),
  ranges   = list(Sepal.Length = c(4, 8), Petal.Width = c(0, 3)),
  domains  = list(Species = levels(iris$Species)),
  rules    = list(petal_smaller = ~ Petal.Length < Sepal.Length)
)
spec
#> 
#> ── Data quality specification ──
#> 
#> • required columns: "Sepal.Length" and "Species"
#> • domains: "Species"
#> • ranges: "Sepal.Length" and "Petal.Width"
#> • rules: "petal_smaller"
#> 
#> ℹ Maximum record age 3652.5 days, assessed as of "2026-10-20".

scored <- dg_assess(iris, spec = spec, verbose = FALSE)
scored$scores
#>     accuracy completeness  consistency  credibility  currentness 
#>         1.00         1.00         0.75           NA           NA
scored$measures[scored$measures$applicable,
                c("code", "value", "a", "b")]
#>             code value   a   b
#> 5      EXAC_SINT  1.00 150 150
#> 7       RAN_EXAC  1.00 300 300
#> 1       COMP_REG  1.00 150 150
#> 2   COMP_VAL_ESP  1.00 300 300
#> 4  FAL_COMP_FICH  1.00 150 150
#> 9      RIES_INCO  0.25   1   4
#> 10    CONS_SEMAN  1.00 150 150
#> 11     CONS_FORM  1.00 150 150

Accuracy is now measurable, where without a spec it was NA. The note column says what each remaining measure is still waiting for:

waiting <- scored$measures[!scored$measures$applicable, c("code", "note")]
waiting
#>            code                                                note
#> 6    EXAC_SEMAN         needs spec$reference and spec$reference_key
#> 3     COMP_FICH                         needs spec$expected_records
#> 8       INT_REF                             needs spec$foreign_keys
#> 12 CRED_VAL_DAT needs spec$credible_ranges or spec$credible_domains
#> 13    CRED_FUEN    needs spec$source_trust and spec$trusted_sources
#> 14     FREC_ACT                          needs spec$update_interval
#> 15     CONV_ACT                                     no date columns

Plausibility is not accuracy

One number sits outside the model deliberately. plausibility flags outliers against an interval derived from the data, where every ISO accuracy property needs one that was declared. An outlier need not be wrong, and a wrong value can sit exactly on the mean – so it is reported separately and left out of the overall score.

report$plausibility
#> [1] 0.9983333
report$scores[["accuracy"]]   # NA: nothing was declared
#> [1] NA

It is still the most useful single number in the report for data that arrives with no documentation, which is most open data.

summary() gives the same content as tables:

summary(report)$columns
#>         column        type n_missing rate n_outliers       vif
#> 1 Sepal.Length     Numeric         0    0          0  7.072722
#> 2  Sepal.Width     Numeric         0    0          1  2.100872
#> 3 Petal.Length     Numeric         0    0          0 31.261498
#> 4  Petal.Width     Numeric         0    0          0 16.090175
#> 5      Species Categorical         0    0         NA        NA

Assessing many data sets at once

as.data.frame() collapses a report to one row, which is what makes a corpus sweep a two-liner:

datasets <- list(iris = iris, mtcars = mtcars, airquality = airquality)
sweep <- do.call(rbind, lapply(names(datasets), function(nm) {
  row <- as.data.frame(dg_assess(datasets[[nm]], verbose = FALSE))
  row$source <- nm
  row
}))
sweep[, c("source", "rows", "completeness", "consistency", "overall",
          "plausibility")]
#>       source rows completeness consistency   overall plausibility
#> 1       iris  150      1.00000   0.6250000 0.8125000    0.9983333
#> 2     mtcars   32      1.00000   0.2727273 0.6363636    0.9971591
#> 3 airquality  153      0.83878   1.0000000 0.9193900    0.9977117

Figures

dg_plots() returns a named list of ggplot objects. Every figure is built from statistics computed during the assessment, so the plot objects never carry the raw observations.

figures <- dg_plots(report)
names(figures)
#> [1] "scores"       "missing"      "correlation"  "outliers"     "distribution"
figures$scores

figures$correlation

figures$distribution

Use dg_save_plots(report, "figures/") to write them all to disk.

Reading from a file

path <- tempfile(fileext = ".csv")
write.csv(mtcars, path, row.names = FALSE)
dg_assess(path = path, verbose = FALSE)$scores
#>     accuracy completeness  consistency  credibility  currentness 
#>           NA    1.0000000    0.2727273           NA           NA
unlink(path)

.csv, .tsv and .txt are read with base R. .xls and .xlsx need the suggested readxl package.

Helpers stand alone

Each of the checks is exported and usable on its own:

detect_data_types(data.frame(id = c("a", "b", "c"), n = c(1.5, 2.5, 3.5)))
#>        id         n 
#>      "ID" "Numeric"
dg_vif(iris)
#> Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
#>     7.072722     2.100872    31.261498    16.090175
analyze_distribution(iris)
#>         column   n   skewness excess_kurtosis    shapiro_p shapiro_n normal
#> 1 Sepal.Length 150  0.3086407      -0.6058125 1.018116e-02       150  FALSE
#> 2  Sepal.Width 150  0.3126147       0.1387047 1.011543e-01       150   TRUE
#> 3 Petal.Length 150 -0.2694109      -1.4168574 7.412263e-10       150  FALSE
#> 4  Petal.Width 150 -0.1009166      -1.3581792 1.680465e-08       150  FALSE