MyNixOS website logo
Description

Pipeline Tools Inspired by 'GNU Make'.

A suite of tools for transforming an existing workflow into a self-documenting pipeline with very minimal upfront costs. Segments of the pipeline are specified in much the same way a 'Make' rule is, by declaring an executable recipe (which might be an R script), along with the corresponding targets and dependencies. When the entire pipeline is run through, only those recipes that need to be executed will be. Meanwhile, execution metadata is captured behind the scenes for later inspection.

makepipe

Codecov testcoverage CRANstatus R-CMD-check

The goal of makepipe is to allow for the construction of make-like pipelines in R with very minimal overheads. In contrast to targets (and its predecessor drake) which offers an opinionated pipeline framework that demands highly functionalised code, makepipe is easy-going, being adaptable to a wide range of data science workflows.

A minimal example can be found here: https://github.com/kinto-b/makepipe_example

Installation

You can install the released version of makepipe from CRAN with:

install.packages("makepipe")

And the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("kinto-b/makepipe")

Building a pipeline

To construct a pipeline, one simply needs to chain together a number of make_with_*() statements. When the pipeline is run through, each make_with_*() block is evaluated if and only if the targets are out-of-date with respect to the dependencies (and source file). But, whether or not the block is evaluated, a segment will be added to the Pipeline object behind the scenes. At the end of the script, once the entire pipeline has been run through, one can display the accumulated Pipeline object to produce a flow-chart visualisation of the pipeline. For example:

make_with_source(
  note = "Clean raw survey data and do derivations",
  source = "one.R",
  targets = "data/1 data.Rds",
  dependencies = c("data/raw.Rds", "lookup/concordance.csv")
)

make_with_recipe(
  label = "Merge it!",
  note = "Merge demographic variables from population data into survey data",
  recipe = {
    dat <- readRDS("data/1 data.Rds")
    pop <- readRDS("data/pop.Rds")
    merged_dat <- merge(dat, pop, by = "id")
    saveRDS(merged_dat, "data/2_data.Rds")
  },
  targets = c("data/2 data.Rds"),
  dependencies = c("data/1 data.Rds", "data/pop.Rds")
)

make_with_source(
  note = "Convert data from 'wide' to 'long' format",
  source = "three.R",
  targets = "data/3 data.Rds",
  dependencies = "data/2 data.Rds"
)

show_pipeline()

We can also get an interactive visNetwork widget:

show_pipeline(as = "visnetwork")

Or a text summary (which can be saved to a .md file),

show_pipeline(as = "text")

#> # Pipeline
#> 
#> ## one.R
#> 
#> Clean raw survey data and do derivations
#> 
#> * Source: 'one.R'
#> * Targets: 'data/1 data.Rds'
#> * File dependencies: 'data/raw.Rds', 'lookup/concordance.csv'
#> * Executed: FALSE
#> * Environment: 0x0000015399acfeb8
#> 
#> ## Merge it!
#> 
#> Merge demographic variables from population data into survey data
#> 
#> * Recipe: 
#> 
#> {
#>     dat <- readRDS("data/1 data.Rds")
#>     pop <- readRDS("data/pop.Rds")
#>     saveRDS(dat, "data/2_data.Rds")
#> }
#> 
#> * Targets: 'data/2 data.Rds'
#> * File dependencies: 'data/1 data.Rds', 'data/pop.Rds'
#> * Executed: TRUE
#> * Execution time: 0.00103879 secs
#> * Result: 0 object(s)
#> * Environment: 0x0000015390c6c568
#> 
#> ## three.R
#> 
#> Convert data from 'wide' to 'long' format
#> 
#> * Source: 'three.R'
#> * Targets: 'data/3 data.Rds'
#> * File dependencies: 'data/2 data.Rds'
#> * Executed: FALSE
#> * Environment: 0x00000153928570f8

Once you’ve constructed a pipeline, you can ‘clean’ it (i.e. delete all registered targets):

p <- get_pipeline()
p$clean()

Then, when you look again at the visualisation, the target nodes will be red not green since they’re out-of-date:

show_pipeline()

And then you can ‘rebuild’ to re-execute the entire pipeline and re-create the cleaned targets:

p <- get_pipeline()
p$build()

Another way to build a pipeline is to add a roxygen header into your .R scripts containing a special @makepipe tag along with the @targets, @dependencies, and so on. For example, at the top of script one.R you might have

#'@title Load
#'@description Clean raw survey data and do derivations
#'@dependencies "data/raw.Rds", "lookup/concordance.csv"
#'@targets "data/1 data.Rds"
#'@makepipe
NULL

You can then call make_with_dir(), which will construct a pipeline using all the scripts it finds in the provided directory containing the @makepipe tag.

Metadata

Version

0.2.1

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows