LoginSign Up
← Back to all sparks
S

sparklyr

ANALYTICS
Velocity0.0

R interface to Apache Spark for distributed data manipulation and machine learning

github.com/rstudio/sparklyr ↗

sparklyr now spends its releases absorbing dbplyr changes and feeding pysparklyr

sparkdatabricksdbplyr-compatibilitymaintenancebackend-refactor
◆Current state
sparklyr connects R to Spark, and almost nothing in this window originates inside the package. Releases restore compatibility after dbplyr changes its SQL generation, adapt to Spark 4.0 and to R 4.4's version-comparison changes, and convert functions into S3 methods so pysparklyr can supply its own implementations.
◆Where it's heading
Two dependencies set the agenda. dbplyr repeatedly changes identifier quoting and lazy-table internals, and each change costs sparklyr a release. Meanwhile the package is being hollowed into a backend: ml_fit(), spark_apply(), spark_write_delta() and now tune_grid_spark() exist as methods so that pysparklyr, the Databricks Connect path, can override them. Dependency removal - tibble, rappdirs, digest - runs alongside as the package slims down.
◆Prediction
Expect the next releases to continue tracking dbplyr and Spark versions, and more functions to be converted to methods as functionality shifts toward pysparklyr; new capability arriving in sparklyr itself looks unlikely.

◆Recent moves

  1. 1mo ago

    Restores compatibility after dbplyr changed Hive quoting

    dbplyr 2.6.0 switched the Hive backend to ANSI double-quoted identifiers and rebuilt the lazy table source, so sparklyr now renders SQL through the Spark SQL backend and pulls connections from lazy tables the new way. The clearest single instance of this package's release cadence being set by dbplyr.

    View source ↗
  2. 3mo ago

    Adds tune_grid_spark() for pysparklyr to implement

    A method added not to be used here but to be filled in by pysparklyr - the same pattern as ml_fit() and spark_apply() before it. sparklyr increasingly defines the interface while the Databricks Connect path supplies the behavior.

    View source ↗
  3. 10mo ago

    Fixes lazy-table field lookup and a name collision

    Three small internal fixes, including avoiding a call to a function named unknown in case another package defines one. Housekeeping with no user-visible surface.

    View source ↗
  4. 1y ago

    Catches up with released Spark 4.0; ml_load() reads via Spark

    Adapts to what changed in Spark 4.0 between preview and release while restoring Spark 2.4 with Scala 2.11 at the other end of the range. ml_load() now reads model metadata through Spark rather than R, which is what makes it work against mounted and non-local file protocols.

    View source ↗
  5. 2y ago

    Databricks autoloader streaming ingestion; R 4.4 fixes

    Adds streaming ingestion of files through Databricks autoloader via stream_read_cloudfiles, the one genuinely new capability in this window. The rest is R 4.4.0 fallout, where version-comparison functions stopped accepting the inputs sparklyr was giving them.

    View source ↗
  6. 2y ago

    Drops tibble and rappdirs; retires Spark 2.3 JARs

    Slimming and pruning in one release - tibble and rappdirs imports removed, Java folders for Spark 2.3 and below deleted, JARs recompiled from 2.4 up. Converting spark_apply() to a method is the early move in the pysparklyr backend pattern.

    View source ↗