MyNixOS website logo
Description

Batch Experiments for 'mlr3'.

Extends the 'mlr3' package with a connector to the package 'batchtools'. This allows to run large-scale benchmark experiments on scheduled high-performance computing clusters.

mlr3batchmark

r-cmd-check CRANstatus StackOverflow Mattermost

A connector between mlr3 and batchtools. This allows to run large-scale benchmark experiments on scheduled high-performance computing clusters.

The package comes with two core functions for switching between mlr3 and batchtools to perform a benchmark:

  • After creating a design object (as required for mlr3’s benchmark() function), instead of benchmark() call batchmark() which populates an ExperimentRegistry for the computational jobs of the benchmark. You are now in the world of batchtools where you can selectively submit jobs with different resources, monitor the progress or resubmit as needed.
  • After the computations are finished, collect the results with reduceResultsBatchmark() to return to mlr3. The resulting object is a regular BenchmarkResult.

Example

library("mlr3")
library("batchtools")
library("mlr3batchmark")
tasks = tsks(c("iris", "sonar"))
learners = lrns(c("classif.featureless", "classif.rpart"))
resamplings = rsmp("cv", folds = 3)

design = benchmark_grid(
  tasks = tasks,
  learners = learners,
  resamplings = resamplings
)

reg = makeExperimentRegistry(NA)
## No readable configuration file found

## Created registry in '/tmp/Rtmp8DlMZQ/registry704553adf7a88' using cluster functions 'Interactive'
ids = batchmark(design, reg = reg)
## Adding algorithm 'run_learner'

## Adding problem 'b39ef23a66b1f1ee'

## Exporting new objects: '5ec484de3f93431b' ...

## Exporting new objects: '7c35d835f3dfae37' ...

## Exporting new objects: '70dd22724e5c724d' ...

## Adding 6 experiments ('b39ef23a66b1f1ee'[1] x 'run_learner'[2] x repls[3]) ...

## Adding problem '76c4fc7a533d41b7'

## Exporting new objects: 'b209de197d6cbe75' ...

## Adding 6 experiments ('76c4fc7a533d41b7'[1] x 'run_learner'[2] x repls[3]) ...
submitJobs()
## Submitting 12 jobs in 12 chunks using cluster functions 'Interactive' ...
getStatus()
## Status for 12 jobs at 2023-11-13 19:32:20:
##   Submitted    : 12 (100.0%)
##   -- Queued    :  0 (  0.0%)
##   -- Started   : 12 (100.0%)
##   ---- Running :  0 (  0.0%)
##   ---- Done    : 12 (100.0%)
##   ---- Error   :  0 (  0.0%)
##   ---- Expired :  0 (  0.0%)
reduceResultsBatchmark()
## <BenchmarkResult> of 12 rows with 4 resampling runs
##  nr task_id          learner_id resampling_id iters warnings errors
##   1    iris classif.featureless            cv     3        0      0
##   2    iris       classif.rpart            cv     3        0      0
##   3   sonar classif.featureless            cv     3        0      0
##   4   sonar       classif.rpart            cv     3        0      0

Resources

  • The Large-Scale Benchmarking chapter of the mlr3 book.
Metadata

Version

0.1.1

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows