LoginSign Up
← Back to all sparks
H

hubValidations

INFRA · APIS
Velocity0.0

Testing framework for hubverse hub validations

github.com/hubverse-org/hubValidations ↗

Submission validation for forecast hubs, rebuilt around which file each check belongs to.

validationepidemiologyhubverser-packagegithub-automation
◆Current state
hubValidations checks submissions to collaborative forecasting hubs — file names, schemas, task-id combinations, sample structure — and runs inside pull-request automation on the hub repository. The package crossed a 2.0.0 boundary that reorganised how multi-file validation results are represented, and the releases since have been narrow additions to sample checking and fixes to the GitHub API integration. Its surface is large, with individual named checks as the unit of extension.
◆Where it's heading
The architectural work is done and the package has returned to incremental check-writing, particularly around samples, where model tasks can carry independent configurations. A parallel thread makes validation output more communicative: informational warnings that do not fail a submission, prominent display of config-file changes, and clearer errors where a cryptic dplyr failure used to surface. The GitHub integration keeps producing small defects, being the one part not exercised by ordinary local use.
◆Prediction
Expect further sample-related checks and continued refinement of what validation output communicates to hub maintainers, rather than another architectural change so soon after 2.0.0.

◆Recent moves

  1. 1mo ago

    Pagination progress message no longer leaks into PR validation output

    Passes .progress = FALSE to the underlying gh call so pagination progress text stops appearing in validate_pr() and validate_target_pr() output. Cosmetic, but it affects what hub maintainers see in an automated PR comment. One more entry in the run of small GitHub-integration defects.

    View source ↗
  2. 4mo ago

    New check enforces sample independence across model tasks

    Adds check_tbl_spl_mt_unique, validating that an individual sample output_type_id does not span multiple model tasks, since different model tasks can carry different sample configurations. It runs ahead of the other sample checks and fails early, and a companion change replaces a cryptic dplyr crash with an informative message for the same condition. Typical of the post-2.0.0 pattern of adding one well-scoped check at a time.

    View source ↗
  3. 5mo ago

    Multi-file validation results become hierarchical (breaking)

    ⚡ SPARK

    The structural release of this window: validation results for multi-file operations stop being a flat list with numerically suffixed duplicate names and become a hierarchy organised by file. It changes how any downstream automation reads validation output, and it is the change every subsequent release builds on. The version number is doing real work here rather than marking maturity.

    View source ↗
  4. 7mo ago

    Validation warnings separated from validation failures

    Introduces infrastructure for informational messages that do not affect whether validation passes, displayed prominently at the top of print output, with check-level warnings available inline and a show_warnings parameter on check_for_errors(). The motivating case is a pull request modifying hub config files, where maintainers need alerting without the submission being rejected. It establishes a middle ground between silence and failure that the check-heavy design previously lacked.

    View source ↗
  5. 9mo ago

    validate_pr() handles pull requests with more than 30 files

    Fixes a GitHub API pagination limit that silently capped retrieved PR files at 30. Narrow, but the failure mode was under-validation of large submissions rather than an error. Another instance of the GitHub integration being the least-exercised part of the package.

    View source ↗
  6. 10mo ago

    target_validations class and hive-partition path utilities

    Adds a target_validations subclass for validating target or truth data, with constructor, coercion and combine methods, plus utilities for extracting key-value pairs from hive-partitioned file paths. Print output shows full paths rather than basenames, which matters when partitioned files share names across directories. Extends validation beyond model submissions to the observed data hubs score against.

    View source ↗