MyNixOS website logo
Description

Access the 'Spark Catalog' API via 'sparklyr'.

Gain access to the 'Spark Catalog' API making use of the 'sparklyr' API. 'Catalog' <https://spark.apache.org/docs/2.4.3/api/java/org/apache/spark/sql/catalog/Catalog.html> is the interface for managing a metastore (aka metadata catalog) of relational entities (e.g. database(s), tables, functions, table columns and temporary views).

{catalog}

CRANstatus Dependencies CRAN downloads

Overview

{catalog} gives the user access to the Spark Catalog API making use of the {sparklyr} API. Catalog is the interface for managing a metastore (aka metadata catalog) of relational entities (e.g. database(s), tables, functions, table columns and temporary views).

Installation

You can install:

  • the development version from GitHub with
# install.packages("remotes")
remotes::install_github("nathaneastwood/catalog")
  • the latest release from CRAN with
install.packages("catalog")

Usage

{catalog} provides an API matching that of the Catalog API and provides full access to all methods. Below is a small example of some of the functionality.

sc <- sparklyr::spark_connect(master = "local")
mtcars_spark <- sparklyr::copy_to(dest = sc, df = mtcars)

library(catalog)

list_tables(sc)
# # A tibble: 1 × 5
#   name   database description tableType isTemporary
#   <chr>  <chr>    <chr>       <chr>     <lgl>      
# 1 mtcars <NA>     <NA>        TEMPORARY TRUE

list_columns(sc, "mtcars")
# # A tibble: 11 × 6
#    name  description dataType nullable isPartition isBucket
#    <chr> <chr>       <chr>    <lgl>    <lgl>       <lgl>   
#  1 mpg   <NA>        double   TRUE     FALSE       FALSE   
#  2 cyl   <NA>        double   TRUE     FALSE       FALSE   
#  3 disp  <NA>        double   TRUE     FALSE       FALSE   
#  4 hp    <NA>        double   TRUE     FALSE       FALSE   
#  5 drat  <NA>        double   TRUE     FALSE       FALSE   
#  6 wt    <NA>        double   TRUE     FALSE       FALSE   
#  7 qsec  <NA>        double   TRUE     FALSE       FALSE   
#  8 vs    <NA>        double   TRUE     FALSE       FALSE   
#  9 am    <NA>        double   TRUE     FALSE       FALSE   
# 10 gear  <NA>        double   TRUE     FALSE       FALSE   
# 11 carb  <NA>        double   TRUE     FALSE       FALSE

list_functions(sc)
# # A tibble: 349 × 5
#    name  database description className                                  isTem…¹
#    <chr> <chr>    <chr>       <chr>                                      <lgl>  
#  1 !     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  2 %     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  3 &     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  4 *     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  5 +     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  6 -     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  7 /     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  8 <     <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
#  9 <=    <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
# 10 <=>   <NA>     <NA>        org.apache.spark.sql.catalyst.expressions… TRUE   
# # … with 339 more rows, and abbreviated variable name ¹​isTemporary
# # ℹ Use `print(n = ...)` to see more rows

drop_temp_view(sc, "mtcars")
# [1] TRUE

For more information, please refer to the package website.

Metadata

Version

0.1.1

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows