Package {datagrade}

datagrade: scored data quality assessment for open data sets

Description

One call, [dg_assess()], profiles a data set and scores it on the five inherent characteristics of the ISO/IEC 25012 data quality model — accuracy, completeness, consistency, credibility and currentness — through the fifteen quality properties beneath them, each measured as a ratio 'A / B' over counted data items in the form ISO/IEC 25024 measurement functions take. It returns a structured 'dg_report' object with tables and figures rather than console text alone.

Details

Six of the fifteen properties are computable from a table alone; the rest need a [dg_spec()] declaring what the data is supposed to look like, because their denominator counts the items for which an expectation exists. Without one they report as not applicable rather than being approximated.

The scoring framework implements the methodology of Demirhan, O. H. (2024), *Assessing Data Quality Management Issues in Open-Sourced Data Sets with Statistical Methodology*, MSc dissertation, University of Greenwich. Where the package deviates from the dissertation, the deviation is recorded in 'NEWS.md' and in the "Deviations from the dissertation" section of the README.

Author(s)

Maintainer: Omer Haluk Demirhan hlk01dn@gmail.com

Authors:

• Omer Haluk Demirhan hlk01dn@gmail.com

See Also

Useful links:

• https://github.com/ohalukdemirhan/datagrade

• Report bugs at https://github.com/ohalukdemirhan/datagrade/issues

Correlation matrix, optionally per category

Description

Correlation matrix, optionally per category

Usage

analyze_correlation(data, category_column = NULL)

Arguments

Value

A list with the overall correlation matrix and, if requested, one matrix per group.

Examples

analyze_correlation(iris, "Species")$overall

Assess the distribution of every numeric column

Description

Reports skewness, excess kurtosis and a Shapiro-Wilk normality test. The test is limited to 5,000 observations by its own definition, so larger columns are tested on a deterministic subsample and the sample size used is reported.

Usage

analyze_distribution(data, sample_size = 5000L, seed = 1L)

Arguments

Value

A data frame with one row per numeric column.

Examples

analyze_distribution(iris)

Summarise missing-data patterns

Description

Summarise missing-data patterns

Usage

analyze_missing_data_patterns(data, high_threshold = 0.3)

Arguments

Value

A list with a per-column summary data frame, the names of heavily incomplete columns, the count of heavily incomplete rows, and the number of rows that are complete in every column.

Examples

analyze_missing_data_patterns(airquality)$columns

Coerce a quality report to a one-row data frame

Description

Convenient for assessing many files and stacking the results.

Usage

## S3 method for class 'dg_report'
as.data.frame(x, row.names = NULL, optional = FALSE, ...)

Arguments

Value

A one-row data frame.

Consistency score from multicollinearity

Description

Consistency score from multicollinearity

Usage

calculate_consistency_score(data, vif_threshold = 5)

Arguments

Value

A list with the score as a ratio in '[0, 1]', the VIF vector, and the names of the offending columns.

Examples

calculate_consistency_score(iris)$score

Identify columns that could serve as a unique key

Description

Identify columns that could serve as a unique key

Usage

check_unique_identifiers(data)

Arguments

Value

A character vector of candidate key columns, possibly empty.

Normalise text columns without destroying information

Description

Trims whitespace and collapses internal runs of spaces. Case folding and punctuation stripping are opt-in and, when enabled, are Unicode-aware.

Usage

clean_text_columns(data, lowercase = FALSE, strip_punctuation = FALSE)

Arguments

Details

The dissertation's implementation applied 'gsub("[^a-zA-Z0-9 ]", "")' unconditionally, which deleted every accented and non-Latin character, every date separator and every '@' in an e-mail address. Assessment is a diagnosis, not a treatment, so nothing here is destructive by default.

Value

The data frame with its character columns normalised.

Examples

clean_text_columns(data.frame(city = c("  Agri ", "Corum")))

Detect the semantic type of every column

Description

Unlike a plain 'class()' lookup this distinguishes an identifier column from an ordinary numeric or text column, and a low-cardinality text column from free text.

Usage

detect_data_types(data, categorical_max = 20L)

Arguments

Details

The uniqueness test is evaluated **before** the storage-type tests. In the dissertation's pseudocode (Appendix A.3) it came last, which made the '"ID"' branch unreachable: a character key matched 'is.character()' and returned '"Text"' first.

Value

A named character vector, one entry per column, with values '"ID"', '"Numeric"', '"Text"', '"Date"', '"Categorical"' or '"Logical"'.

Examples

detect_data_types(data.frame(id = c("a", "b"), n = 1:2))

Count duplicate rows

Description

Returns a count and a small sample rather than the full set of duplicated rows. On a data set with millions of near-identical rows, materialising every duplicate can be larger than the input itself.

Usage

detect_duplicates(data, columns = NULL, max_examples = 5L)

Arguments

Value

A list with 'n' (count), 'rate' (share of rows) and 'examples'.

Detect outliers in numeric columns

Description

Detect outliers in numeric columns

Usage

detect_outliers(
  data,
  numeric_columns = NULL,
  method = c("z_score", "iqr"),
  category_column = NULL,
  threshold = 3
)

Arguments

Value

A list with the total count, the share of tested values that were flagged, and a per-column breakdown.

Examples

detect_outliers(iris)$total

Assess the quality of a data set

Description

Runs every check in the package and returns a 'dg_report' object scoring the data on the five inherent characteristics of the ISO/IEC 25012 data quality model — accuracy, completeness, consistency, credibility and currentness — through the fifteen quality properties beneath them. Each property is measured as a ratio 'A / B' over counted data items, the form ISO/IEC 25024 measurement functions take.

Usage

dg_assess(
  data = NULL,
  path = NULL,
  spec = NULL,
  outlier_method = c("z_score", "iqr"),
  imputation_method = c("mean", "median", "none"),
  correlation_threshold = 0.5,
  vif_threshold = 5,
  category_column = NULL,
  weights = c(accuracy = 1, completeness = 1, consistency = 1, credibility = 1,
    currentness = 1),
  clean_text = TRUE,
  strip_punctuation = FALSE,
  parse_dates = TRUE,
  check_duplicates = TRUE,
  anomaly_detection = FALSE,
  sample_size = 1e+05,
  seed = 1L,
  keep_data = FALSE,
  verbose = TRUE
)

data_quality_assessment(...)

Arguments

Details

Six of the fifteen properties are computable from a table alone. The other nine have a denominator that counts *the items for which an expectation is declared*, which no data set contains; supply a [dg_spec()] to make them measurable. Without one they are reported as not applicable and dropped from the aggregate, never scored as zero.

A sixth, non-ISO score is reported alongside them. 'plausibility' is the outlier-based measure this package used before it followed the standard: it compares values against an interval *derived* from the data rather than one declared in a specification. That makes it useful and cheap on data with no documentation, but it is not accuracy in the ISO sense — an outlier need not be wrong, and a wrong value can sit exactly on the mean — so it is reported separately and excluded from the overall score.

The function never modifies the caller's data. Text normalisation and imputation are applied to an internal copy used for the statistics, and the imputed frame is returned only when 'keep_data = TRUE'.

'data_quality_assessment()' is the name used in the dissertation and is kept as an alias.

Value

An object of class 'dg_report'. See [print.dg_report()], [summary.dg_report()] and [dg_plots()].

Examples

report <- dg_assess(iris, verbose = FALSE)
report$scores
report$measures[, c("code", "value", "a", "b")]

# Declaring an expectation makes the accuracy measures computable.
spec <- dg_spec(ranges = list(Sepal.Length = c(4, 8)),
                domains = list(Species = levels(iris$Species)))
dg_assess(iris, spec = spec, verbose = FALSE)$scores

Figures for a quality report

Description

Returns a named list of ggplot2 objects. Every figure is built from statistics precomputed during [dg_assess()] — bin counts, five-number summaries, a correlation matrix — so the plot objects never carry the raw observations and rendering cost does not grow with the number of rows.

Usage

dg_plots(x, which = NULL)

## S3 method for class 'dg_report'
plot(x, y, ...)

Arguments

Value

A named list of ggplot objects: 'scores', 'missing', 'correlation', 'outliers', 'distribution'.

Examples

report <- dg_assess(airquality, verbose = FALSE)
figures <- dg_plots(report)
names(figures)

Save every report figure to a directory

Description

Save every report figure to a directory

Usage

dg_save_plots(x, dir, width = 9, height = 5.5, dpi = 150, device = "png")

Arguments

Value

The file paths, invisibly.

Examples

report <- dg_assess(iris, verbose = FALSE)
out <- file.path(tempdir(), "datagrade-figures")
paths <- dg_save_plots(report, out)
basename(paths)
unlink(out, recursive = TRUE)

Declare what the data is supposed to look like

Description

Nine of the fifteen ISO/IEC 25012 quality properties cannot be computed from a table alone, because their denominator counts *the data items for which an expectation is defined* — and an expectation is something a person declares, not something a data set contains. 'dg_spec()' is where those declarations go. Pass the result to [dg_assess()] as 'spec'.

Usage

dg_spec(
  required = NULL,
  expected_records = NULL,
  null_equivalents = NULL,
  patterns = NULL,
  domains = NULL,
  ranges = NULL,
  reference = NULL,
  reference_key = NULL,
  foreign_keys = NULL,
  rules = NULL,
  formats = NULL,
  credible_ranges = NULL,
  credible_domains = NULL,
  source_trust = NULL,
  trusted_sources = NULL,
  update_column = NULL,
  update_interval = NULL,
  max_age = 3652.5,
  as_of = Sys.Date()
)

Arguments

Details

Every argument is optional. Each one you supply switches on the measures that depend on it; each one you leave out leaves those measures reported as *not applicable* rather than guessed at. Supplying nothing is legal and gives the same six zero-configuration measures [dg_assess()] computes on its own.

Value

An object of class 'dg_spec'.

Examples

spec <- dg_spec(
  required = c("Sepal.Length", "Species"),
  ranges   = list(Sepal.Length = c(4, 8)),
  domains  = list(Species = c("setosa", "versicolor", "virginica"))
)
spec
dg_assess(iris, spec = spec, verbose = FALSE)$scores

Variance inflation factors

Description

Computed from the diagonal of the inverted correlation matrix, which is algebraically identical to '1 / (1 - R^2_j)' — the definition given in the dissertation — but costs 'O(p^3)' in the number of columns and is independent of the number of rows. Fitting 'p' auxiliary regressions instead, as the original implementation did through a dummy response, is 'O(n p^2)' and does not finish in reasonable time on millions of rows.

Usage

dg_vif(data)

Arguments

Value

A named numeric vector of VIF values. Perfectly collinear columns are reported as 'Inf'.

Examples

dg_vif(iris)

Flag rows whose variable relationships look anomalous

Description

Fits a random forest predicting the first numeric column from the others and flags rows whose residual exceeds 'sd_threshold' standard deviations. The model is always fitted on a deterministic subsample, then applied to the full data, so cost is bounded by 'sample_size' rather than by the input size.

Usage

flag_anomalies_in_relationships(
  data,
  target = NULL,
  sample_size = 20000L,
  sd_threshold = 2,
  ntree = 100L,
  seed = 1L
)

Arguments

Details

Requires the suggested randomForest package.

Value

A list with the number of anomalies, their share, and example rows; or 'NULL' when the input has too few numeric columns.

Flag redundant columns by pairwise correlation

Description

Flag redundant columns by pairwise correlation

Usage

flag_redundant_columns(data, correlation_threshold = 0.5)

Arguments

Value

A list with the correlation matrix, a data frame of offending pairs, and the columns proposed for removal.

Examples

flag_redundant_columns(iris, 0.9)$columns

Impute missing numeric values

Description

Impute missing numeric values

Usage

impute_missing_values(
  data,
  method = c("mean", "median", "none"),
  group_by = NULL
)

Arguments

Value

The data frame with numeric 'NA's filled.

Examples

colSums(is.na(impute_missing_values(airquality, "median")))

Print a quality report

Description

Print a quality report

Usage

## S3 method for class 'dg_report'
print(x, ...)

Arguments

Value

'x', invisibly.

Tabular summary of a quality report

Description

Tabular summary of a quality report

Usage

## S3 method for class 'dg_report'
summary(object, ...)

Arguments

Value

A list of data frames: 'scores', 'measures', 'columns', 'redundant_pairs' and 'distribution'.

A recessive ggplot2 theme for quality-report figures

Description

Grid and axes are deliberately low-contrast so the marks carry the reading.

Usage

theme_dg(base_size = 12)

Arguments

Value

A [ggplot2::theme()] object.

Check date columns for impossible and stale values

Description

Check date columns for impossible and stale values

Usage

validate_dates(data, stale_years = 10)

Arguments

Value

A data frame with one row per date column: number of future dates, number of stale dates, and the observed range.