LoginSign Up
← Back to all sparks
F

filearray

ANALYTICS
Velocity0.0

File-Backed Array for Out-of-Memory Computation

github.com/dipterix/filearray ↗

The on-disk array layer under RAVE spends its releases hunting segfaults.

on-disk-arraysmemory-safetyc++performanceinfrastructure
◆Current state
filearray stores large arrays on disk and reads them back with little memory overhead, serving as the storage substrate for the RAVE intracranial EEG stack. The 0.2.2 release fixes out-of-bound indexing that caused segfaults along certain margins and an ASAN-flagged signed integer overflow in the load path. The user-facing API has been stable since 0.1.6.
◆Where it's heading
This is infrastructure whose release history reads as a memory-safety log: unprotected C++ variables, buffer sizes exceeding array length, allocations one byte short, endianness on big-endian platforms, and now out-of-bound margins caught by sanitizers. The one sustained feature direction is reducing the cost of operating on arrays too large for memory — lazy operator evaluation through a proxy class, fmap-style application, and marginal collapse. Portability work has steadily removed hard requirements, dropping the C++11 declaration and swapping OpenMP for TinyThreads to get parallelism on macOS.
◆Prediction
Expect continued sanitizer-driven patches rather than new interfaces; the three-year gap before 0.2.2 suggests releases now arrive only when a crash or a CRAN check demands one.

◆Recent moves

  1. 2mo ago

    Segfault on out-of-bound margins fixed

    Fixes out-of-bound indexing that crashed R along certain margins, plus a signed integer overflow in load.cpp caught by ASAN. Arriving three years after the previous release, it reads as the package being woken up by sanitizer reports rather than by feature demand.

    View source ↗
  2. 3y ago

    Lazy operator evaluation and macOS parallelism

    Adds an array proxy class so simple operators evaluate lazily, an as_filearray method with optional float support, and user-defined temporary paths, while swapping OpenMP for TinyThreads to get parallel execution working on macOS. The most substantial release in this feed and the last time the interface meaningfully moved.

    View source ↗
  3. 3y ago

    Sequential read bug in fmap corrected

    Fixes undefined behavior when the buffer size exceeded the array length in fmap and fmap2, with the maintainer noting the impact is minor and the fix passes a valgrind check.

    View source ↗
  4. 4y ago

    Partition limit removed by opening files on demand

    Stops opening every partition's file descriptor up front, which had hit operating system connection limits on large writes, trading some write throughput for robustness. Also fixes an allocation one byte short of the request that could crash R, and adds filearray_load_or_create for replacing arrays whose headers no longer match.

    View source ↗
  5. 4y ago

    Header signatures and symbolic link detection

    Adds header signature methods and filearray_checkload for validating an array before use, detects whether symbolic links actually work, and lets filearray_bind reuse a cache when signatures agree. Groundwork for treating on-disk arrays as durable artifacts rather than scratch files.

    View source ↗
  6. 4y ago

    Flush timing left to the operating system

    Stops forcing a flush when saving and allows arrays to expand along the partition margin, plus a dimension-name fix. Low-level tuning with no interface change.

    View source ↗