Description
Provides a 'PySpark' Back-End for the 'sparklyr' Package.
Description
It enables 'sparklyr' to integrate with 'Spark Connect', and 'Databricks Connect' by providing a wrapper over the 'PySpark' 'python' library.
README.md
pysparklyr
Integrates sparklyr with PySpark and Databricks. The main reason of this package is because the new Spark and Databricks Connect connection method does not work with standard sparklyr integration.
Installing
To install the version in CRAN use:
install.packages("pysparklyr")
To get the development version from GitHub use:
remotes::install_github("mlverse/pysparklyr")
Using
To learn how to use, please visit the Spark / Databricks Connect article, available in the official sparklyr website: Spark Connect, and Databricks Connect v2