Classify, compare, and resume

Gaurav Sood

A domain-list lookup answers a different question from inspecting a page. Choose an evidence source first, then retain its identity with the result. The examples below use local fixtures so the vignette can run without network access or keys.

Preserve every membership

path <- tempfile(fileext = ".csv")
write.csv(data.frame(
  domain = c("example.com", "example.com", "university.example"),
  category = c("news", "shopping", "education"),
  usage = c("black", "black", "white")
), path, row.names = FALSE)
results <- ut1_cat(c("https://example.com/article", "missing.example"), use_file = path)
results[c("input_id", "input", "provider", "status", "category")]
## # A tibble: 2 × 5
##   input_id input                       provider status   category
##      <int> <chr>                       <chr>    <chr>    <chr>   
## 1        1 https://example.com/article ut1      ok       <NA>    
## 2        2 missing.example             ut1      no_match <NA>
results$labels[[1]]
## # A tibble: 2 × 3
##   category score vendor
##   <chr>    <dbl> <chr> 
## 1 news        NA <NA>  
## 2 shopping    NA <NA>
fetch_report(results)
## # A tibble: 1 × 8
##       n    ok failed no_match retryable by_stage         by_reason missing  
##   <int> <int>  <int>    <int>     <int> <list>           <list>    <list>   
## 1     2     1      0        1         0 <tibble [0 × 2]> <tibble>  <chr [0]>

The first domain has two memberships, so no single category is selected. The second has no match, which is different from a failed download. UT1 usage and raw records remain in details. The fixture has no verified metadata, so its dates are unknown.

Preserve identifiers when resuming

pending <- which(results$retryable)
retried <- ut1_cat(results$input[pending], use_file = path)
retried$input_id <- results$input_id[pending]
results[pending, ] <- retried
stopifnot(nrow(results) == 2L, nrow(retried) == 0L)
unlink(path)

For a paid provider, save completed chunks with saveRDS() and retry only failed inputs using the same provider, model, and vocabulary. Keep original identifiers when combining chunks. A result’s no_match state does not call for a retry.

Distinguish page state from content classification

page_signals("<html><body>This domain is for sale. Inquire now.</body></html>")
## # A tibble: 1 × 8
##   page_state blocked block_vendor block_reason parked unavailable thin  n_tokens
##   <chr>      <lgl>   <chr>        <chr>        <lgl>  <lgl>       <lgl>    <int>
## 1 parked     FALSE   <NA>         <NA>         TRUE   FALSE       TRUE         7
news_url_signals(c("https://example.com/sport", "https://example.com/politics"))
## # A tibble: 2 × 4
##   input_id input                        soft_news hard_news
##      <int> <chr>                        <lgl>     <lgl>    
## 1        1 https://example.com/sport    TRUE      FALSE    
## 2        2 https://example.com/politics FALSE     TRUE

Page-state detection is heuristic. URL keyword flags are neither mutually exclusive nor a substitute for reading the page. The content classifier uses Python’s own HTML extraction and its pinned model weights. Its documented quality limitations still apply.

reticulate::py_require("piedomains==0.15.0", python_version = ">=3.12")
pages <- collect_content("https://example.com/article")
classified <- pie_cat(pages = pages)
classified$details[[1]][c("fetched_at", "source", "snapshot_timestamp", "fetch_status")]

Historical collection uses the exact URL and the most recent capture on or before archive_date. Current collection does not use archived fallback unless requested.