LoginSign Up
← Back to all sparks
S

StreamCatTools

ANALYTICS
Velocity0.0

StreamCatTools: Tools for Working with StreamCat and LakeCat Data

github.com/usepa/streamcattools ↗

StreamCatTools is quietly moving off web services and onto cloud-native GeoParquet

repawatershed datageoparquetduckdbapi client
◆Current state
StreamCatTools is the R client for the US EPA's StreamCat and LakeCat watershed metrics datasets. Through 2025 it worked almost entirely against the StreamCat web API, and much of its release history is about making those HTTP calls survive real conditions — POST bodies for long COMID lists, tryCatch around service outages, tests that skip when the service is down. 0.11.0 in May 2026 introduces a different data path.
◆Where it's heading
The newest release adds lc_get_watershed(), which returns a lake watershed as an sf feature by reading an S3-hosted, HUC2-partitioned GeoParquet dataset with DuckDB, optional HUC2 filtering, retries and multi-threading. That is a materially different access model from the web service the rest of the package uses. Alongside it, 0.10.0 added National Nutrient Inventory access and nitrogen/phosphorus budget plotting, extending the package past retrieval into presentation.
◆Prediction
If the GeoParquet path proves faster and more reliable than the web service, migrating more retrieval functions onto it is the obvious next move. The entries do not say whether EPA intends to publish the full metric catalogue in that format.

◆Recent moves

  1. 3mo ago

    Lake watersheds read from S3-hosted GeoParquet via DuckDB

    Adds lc_get_watershed(), returning a lake watershed as an sf feature for a LakeCat COMID from an S3-hosted, HUC2-partitioned GeoParquet dataset, using DuckDB with optional HUC2 filtering, retries and multi-threading. A different access model from the web service the rest of the package relies on, and the first sign of a cloud-native data path.

    View source ↗
  2. 6mo ago

    National Nutrient Inventory access and budget plotting

    Adds sc_get_nni() and lc_get_nni() for National Nutrient Inventory data, plus sc_plot() and lc_plot() for nitrogen and phosphorus watershed budgets. The plotting functions move the package past pure retrieval into presenting the metrics it fetches.

    View source ↗
  3. 9mo ago

    Skip tests when web service is down; vignettes converted to articles

    Uses skip_on_cran() to handle web service outages during testing and converts most vignettes to articles with a new Start Here vignette. Test and documentation plumbing.

    View source ↗
  4. 10mo ago

    Graceful failure when web services are unavailable

    Adds tryCatch handling so functions calling web services fail gracefully instead of erroring, addressing a CRAN test error. Defensive work against a dependency the package does not control.

    View source ↗
  5. 10mo ago

    Metric name search and vector arguments for data retrieval

    Adds sc_get_metric_names() and lc_get_metric_names() to make the metric catalogue searchable, and lets sc_get_data() and lc_get_data() take vectors as well as comma-separated strings for metric, state, county and region. Discoverability for a dataset with a large metric list.

    View source ↗
  6. 11mo ago

    Data requests moved to POST body for long COMID lists

    Switches sc_get_data() and lc_get_data() to pass parameters in the POST request body rather than the header, removing the length limit that capped how many COMIDs a single request could carry. A protocol detail that determines whether large watershed queries are possible at all.

    View source ↗