LoginSign Up
← Back to all sparks
C

collapse

ANALYTICS
Velocity0.0

Advanced and Fast Data Transformation

github.com/SebKrantz/collapse ↗

collapse got a JSS paper and a 7x fmean speedup in the same release.

data-transformationperformancesimdgrouped-statisticsfastverser-package
◆Current state
collapse provides fast grouped statistical computing and data transformation for R, built on a C backend with its own grouping, hashing and aggregation primitives. The 2.1.x line is a maintenance and optimization series: SIMD multiple-accumulator work delivering roughly 2x on fsum() and 7x on fmean() for systems without OpenMP, a custom internal unlist() with better attribute preservation, and a steady stream of correctness fixes in collap(), pivot() and roworderv().
◆Where it's heading
The package is consolidating institutionally as much as technically. The repository moved to the fastverse organization with multiple people granted access, the Journal of Statistical Software paper landed as the primary citation, and documentation now includes an AI-generated interactive layer. Technically the focus is the hashing and grouping core — the decision to treat -0 and 0 as equal across funique(), group(), fmatch(), fmode() and their derivatives was made in sync with an equivalent change in Rcpp, and accepted a measured 3% cost to get it. The last release with breaking changes sits outside this six-entry window.
◆Prediction
Expect further targeted performance work on the grouped statistical functions and continued small correctness fixes; the governance move to fastverse suggests contribution volume rather than direction is what the maintainer is managing.

◆Recent moves

  1. 2mo ago

    SIMD accumulators give fmean a 7x speedup without OpenMP

    Multiple-accumulator SIMD optimization lands on the two most-used aggregators, roughly doubling fsum() and giving fmean() about seven times its previous speed on systems without OpenMP — precisely the machines that were worst served before. GRP.default() gains a drop argument for retaining unobserved factor level combinations, and the JSS paper becomes the package's primary citation.

    View source ↗
  2. 7mo ago

    Negative zero now hashes equal to zero across the package

    A correctness change at the core: -0 and 0 are treated as the same value in every hash-based function, from funique() and group() through fmatch() and fmode(), implemented in sync with the equivalent change in Rcpp and accepting a roughly 3% hashing cost. The repository also moves to the fastverse organization with shared access.

    View source ↗
  3. 8mo ago

    collap() no longer double-aggregates external weights

    When grouping was passed externally as columns or a GRP object, collap() aggregated the weights twice — once with the wrong function. A defect that produced plausible numbers rather than an error, which is the kind that survives longest.

    View source ↗
  4. 9mo ago

    Custom unlist() preserves attributes

    A contributed internal replacement for unlist() that keeps attributes base R would discard, with a small speed gain alongside. Low-level groundwork that shows up as fewer surprises in the higher-level functions.

    View source ↗
  5. 0y ago

    Assorted bug fixes

    Three referenced issues closed with no description of what changed. Nothing in the release record indicates user-visible effect.

    View source ↗
  6. 1y ago

    na_insert gains by-reference mode; gsplit and pivot speed up

    A set argument lets na_insert() modify by reference rather than copy, consistent with the package's broader preference for avoiding allocation. Moderate speed gains reach gsplit(), BY() and pivot().

    View source ↗