LoginSign Up
← Back to all sparks
P

pysparklyr

ANALYTICS
Velocity3.8

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect

github.com/mlverse/pysparklyr ↗

Posit's Spark Connect bridge keeps adding backends — and now runs tidymodels tuning on the cluster.

sparkdatabrickssnowflaketidymodelsr-python-interopdistributed-compute
◆Current state
pysparklyr is the Python-backed backend that lets sparklyr talk to Spark Connect, Databricks Connect, and now Snowflake, handling the reticulate environment, authentication, and Arrow configuration so R users mostly do not have to. The 0.2.x line has widened it well past a connectivity shim: 0.2.0 brought the Spark 4.0 ML function family and Snowpark Connect, and 0.2.2 added tune_grid_spark() so a tidymodels tuning grid executes inside a Spark Connect cluster. Authentication has become a first-class concern, with Snowflake's native authenticators, connections.toml discovery, and Posit Connect viewer credentials all supported.
◆Where it's heading
Two directions are running at once. Horizontally, the package is becoming backend-plural — what started as Databricks-and-Spark now covers Snowflake through Snowpark Connect, with credential handling generalized per platform rather than special-cased. Vertically, it is climbing from data manipulation toward modeling: distributed ML functions in 0.2.0, distributed tuning in 0.2.2. A persistent third thread is absorbing upstream churn — Pandas 3.0 conversion, sparklyr 1.9.5 and dbplyr 2.6.0 restructuring the tbl source slot, reticulate's changing environment management.
◆Prediction
With tuning distributed and the Spark 4.0 ML surface in place, the unfinished edge is the rest of the tidymodels workflow — expect fitting and resampling paths to follow tune_grid_spark() onto the cluster.

◆Recent moves

  1. 28d ago

    tune_grid_spark() runs tidymodels tuning on Spark Connect

    ⚡ SPARK

    Adds tune_grid_spark(), pushing a tidymodels tuning grid into a Spark Connect cluster, plus full native authenticator support for Snowflake via its Python SDK including connections.toml credentials. Continues the climb from moving data to running models, and restores compatibility after sparklyr 1.9.5 and dbplyr 2.6.0 restructured the tbl source slot.

    View source ↗
  2. 6mo ago

    Spark 4.0 ML functions and Snowpark Connect support

    ⚡ SPARK

    Brings the Spark 4.0 ML methodology into the R binding — generalized linear and isotonic regression, GBT classifier and regressor, AFT survival regression, bisecting k-means, decision trees, k-means, and the evaluator family — and adds Snowflake's Snowpark Connect as a connection method. The first release where the package is plainly not Databricks-specific.

    View source ↗
  3. 10mo ago

    Delta writes and a more flexible Python environment picker

    Adds spark_write_delta() and reworks the Databricks connections app so the Python environment is a dropdown that detects and defaults to a virtual environment inside the current RStudio project. Small but aimed at the two things that most often stall a first connection.

    View source ↗
  4. 1y ago

    rpy2 install deferred to first spark_apply() call

    Stops installing rpy2 automatically in uv-based environments, deferring it to py_require() when spark_apply() is actually called, and restores Databricks host name sanitation. Continues a long-running effort to make the Python environment appear only when needed.

    View source ↗
  5. 1y ago

    Databricks serverless compute and SDK-deferred authentication

    Adds support for Databricks serverless interactive compute and defers authentication to the Databricks SDK rather than handling methods itself. Also adopts reticulate's new Python environment management so install_pyspark() and install_databricks() are no longer needed in interactive sessions — a notable reduction in first-run setup.

    View source ↗
  6. 1y ago

    Positron IDE detection and connection-pane fixes

    Adds an IDE check for Positron, stops installing rpy2 by default, and fixes host URL sanitation plus catalog and schema names containing dashes in the Connections Pane. Routine maintenance on the connection experience.

    View source ↗