Batch Processing & Quota Management

Understanding YouTube API quotas

YouTube now tracks ordinary Data API units, search calls, and video-upload calls in separate default buckets. Google Cloud Console remains the source of truth; the package counters only estimate requests made by the current R session.

Quota Costs by Operation

Different operations have different quota costs:

Operation	Default accounting	Example function
Ordinary list request	1 Data API unit	`get_video_details()`
Search request	1 Search Query call	`yt_search()`
Typical write or delete	50 Data API units	`create_playlist()`, `delete_video()`
Caption list, download, insert, update	50, 200, 400, 450 Data API units	Caption functions
Video upload	1 Video Upload call	`upload_video()`

Google’s default allocations are 10,000 Data API units, 100 Search Query calls, and 100 Video Upload calls per day. These defaults can change or be increased for a project.

Tracking Quota Usage

tuber provides built-in quota tracking:

library(tuber)

# Check current quota usage
quota_status <- yt_get_quota_usage()
print(quota_status)

# View details
quota_status[, c("bucket", "quota_used", "quota_limit", "quota_remaining")]
quota_status$reset_time[1] # Midnight Pacific Time

Setting Custom Quota Limits

If you’ve requested a quota increase from Google, update your limit:

# Set a custom quota limit
yt_set_quota_limit(50000, bucket = "data")

# Configure a project-specific search-call allocation
yt_set_quota_limit(200, bucket = "search")

# Reset quota tracking (e.g., after midnight)
yt_reset_quota()

Authentication Strategies

tuber supports two authentication methods, each with different use cases.

API Key Authentication

Best for read-only public data: - Searching videos - Getting public video statistics - Fetching channel information - Reading public comments

# Set up API key (get one from Google Cloud Console)
yt_set_key("YOUR_API_KEY")

# Use API key for read operations
video_stats <- get_video_details(
  video_ids = "dQw4w9WgXcQ",
  auth = "key"
)

channel_stats <- get_channel_details(
  channel_ids = "UCuAXFkgsw1L7xaCfnd5JJOw",
  auth = "key"
)

OAuth2 Authentication

Required for: - Accessing private/unlisted content - Writing data (creating playlists, uploading videos) - Deleting resources - Accessing user’s own channel data

# Set up OAuth2 (opens browser for authentication)
yt_oauth("YOUR_CLIENT_ID", "YOUR_CLIENT_SECRET")

# Write operations require OAuth
create_playlist(
  title = "My New Playlist",
  description = "A collection of favorites",
  status = "private"
)

# Access your own channel's unlisted videos
list_my_videos()

Which Authentication to Use?

Task	API Key	OAuth2
Search public videos	Yes	Yes
Get public video stats	Yes	Yes
Get public channel info	Yes	Yes
Read public comments	Yes	Yes
Access unlisted videos (own)	No	Yes
Access private videos (own)	No	Yes
Create/update playlists	No	Yes
Upload videos	No	Yes
Delete content	No	Yes
Manage comments	No	Yes

Batch Processing

When working with many videos or channels, batch processing is essential for efficiency.

Processing Multiple Videos

# Get details for multiple videos at once
video_ids <- c("dQw4w9WgXcQ", "M7FIvfx5J10", "kJQP7kiw5Fk")

# Single API call for multiple videos (more efficient)
videos <- get_video_details(
  video_ids = video_ids,
  part = c("snippet", "statistics"),
  auth = "key"
)

# Analyze results
head(videos)

Using Bulk Analysis Functions

tuber provides high-level functions for common analysis tasks:

# Comprehensive video analysis
analysis <- bulk_video_analysis(
  video_ids = video_ids,
  include_comments = FALSE,
  auth = "key"
)

# Access results
analysis$video_data    # Detailed video information
analysis$benchmarks    # Performance percentiles
analysis$summary       # Overall statistics

# Channel analysis
channel_analysis <- analyze_channel(
  channel_id = "UCuAXFkgsw1L7xaCfnd5JJOw",
  max_videos = 50,
  auth = "key"
)

# Compare multiple channels
comparison <- compare_channels(
  channel_ids = c("UC1", "UC2", "UC3"),
  metrics = c("subscriber_count", "video_count", "view_count"),
  auth = "key"
)

Pagination Handling

tuber automatically handles pagination for large result sets:

# Request more items than API allows per page (50)
# tuber automatically makes multiple API calls
playlist_items <- list_playlist_items(
  playlist_id = "PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
  max_results = 200,  # Will make 4 API calls
  auth = "key"
)

# Get many comments with automatic pagination
comments <- list_comment_threads(
  video_id = "dQw4w9WgXcQ",
  max_results = 500,  # Will paginate automatically
  auth = "key"
)

Extracting Data from Results

YouTube API responses contain nested JSON. Here’s how to work with them.

Using simplify = TRUE (Default)

Most functions flatten nested data into data frames:

# Get simplified output
videos <- get_video_details(
  video_ids = "dQw4w9WgXcQ",
  simplify = TRUE,
  auth = "key"
)

# Access fields directly
videos$title
videos$viewCount
videos$likeCount
videos$channelTitle

Using simplify = FALSE for Complex Data

When you need the full nested structure:

# Get raw API response
videos_raw <- get_video_details(
  video_ids = "dQw4w9WgXcQ",
  simplify = FALSE,
  auth = "key"
)

# Navigate nested structure
video <- videos_raw$items[[1]]
video$snippet$title
video$snippet$thumbnails$high$url
video$statistics$viewCount
video$contentDetails$duration

Common Field Access Patterns

# Video details
videos$snippet.title          # Title
videos$snippet.description    # Description
videos$statistics.viewCount   # View count
videos$statistics.likeCount   # Like count
videos$contentDetails.duration # Duration (ISO 8601)

# Channel details
channels$snippet.title           # Channel name
channels$statistics.subscriberCount
channels$statistics.videoCount
channels$brandingSettings.channel.description

# Comment extraction
comments$text_display
comments$author_name
comments$like_count

Error Handling & Retries

Using with_retry for Transient Errors

# Automatic retry with exponential backoff
result <- with_retry(
  get_video_details(video_ids = "dQw4w9WgXcQ", auth = "key"),
  max_retries = 3,
  base_delay = 1
)

Handling Quota Exhaustion

# Check before making data API requests
quota <- yt_get_quota_usage()
data_quota <- quota[quota$bucket == "data", ]
if (data_quota$quota_remaining < 100) {
  warning("Low quota! Consider waiting until reset at: ", data_quota$reset_time)
}

# Wrap expensive operations
tryCatch({
  results <- yt_search(term = "R programming", max_results = 50)
}, error = function(e) {
  if (grepl("quota", e$message, ignore.case = TRUE)) {
    message("Quota exceeded. Try again after: ", yt_get_quota_usage()$reset_time)
  }
})

Rate Limiting Best Practices

# Add delays between requests
video_ids <- c("id1", "id2", "id3", "id4", "id5")

results <- lapply(video_ids, function(vid) {
  Sys.sleep(0.5)  # 500ms delay between requests
  get_video_details(video_ids = vid, auth = "key")
})

Caching for Performance

tuber includes built-in caching for frequently accessed data:

# Configure cache
tuber_cache_config(
  enabled = TRUE,
  max_size = 100,
  default_ttl = 3600
)

# Eligible GET requests are cached transparently
cats <- list_video_categories(region_code = "US", auth = "key")
langs <- list_languages(auth = "key")
regions <- list_regions(auth = "key")

# Check cache status
tuber_cache_info()

# Clear cache when needed
tuber_cache_clear()

Practical Examples

Example 1: Analyze a Channel’s Performance

# Full channel analysis
analysis <- analyze_channel(
  channel_id = "UCuAXFkgsw1L7xaCfnd5JJOw",
  max_videos = 100,
  auth = "key"
)

# Summary statistics
cat("Channel:", analysis$channel_info$title, "\n")
cat("Subscribers:", analysis$channel_info$subscriberCount, "\n")
cat("Average views:", analysis$performance_metrics$avg_views_per_video, "\n")
cat("Engagement rate:", analysis$performance_metrics$engagement_rate, "\n")

Example 2: Trending Analysis

# Analyze trending topics
trends <- analyze_trends(
  search_terms = c("machine learning", "AI", "data science"),
  time_period = "month",
  max_results = 25,
  region_code = "US",
  auth = "key"
)

# View trend summary
print(trends$trend_summary)

# Most trending term
best_trend <- trends$trend_summary[1, ]
cat("Top trending:", best_trend$search_term, "\n")
cat("Total views:", best_trend$total_views, "\n")

Example 3: Efficient Video Processing

# Get all videos from a playlist
playlist_videos <- list_playlist_items(
  playlist_id = "PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf",
  max_results = 100,
  auth = "key"
)

# Extract video IDs
video_ids <- playlist_videos$video_id

# Get detailed stats for all videos in one call
video_stats <- get_video_details(
  video_ids = video_ids,
  part = c("statistics", "contentDetails"),
  auth = "key"
)

# Analyze performance
total_views <- sum(as.numeric(video_stats$viewCount), na.rm = TRUE)
avg_duration <- mean(as.numeric(video_stats$duration), na.rm = TRUE)

Troubleshooting

Common Issues

“quotaExceeded” error: - Check quota with yt_get_quota_usage() - Wait until reset_time or request quota increase from Google

“forbidden” error: - Ensure YouTube Data API is enabled in Google Cloud Console - Check API key/OAuth credentials are correct - Verify the resource isn’t private

“videoNotFound” or empty results: - Video may be private, deleted, or region-restricted - Double-check the video/channel ID format

Rate limiting (429 errors): - Add delays with Sys.sleep() between requests - Use with_retry() for automatic backoff

Getting Help

# Check function documentation
?get_video_details
?yt_search
?with_retry

# View package vignettes
browseVignettes("tuber")