Description
Batch Experiments for 'mlr3'.
Description
Extends the 'mlr3' package with a connector to the package 'batchtools'. This allows to run large-scale benchmark experiments on scheduled high-performance computing clusters.
README.md
mlr3batchmark
A connector between mlr3 and batchtools. This allows to run large-scale benchmark experiments on scheduled high-performance computing clusters.
The package comes with two core functions for switching between mlr3
and batchtools
to perform a benchmark:
- After creating a
design
object (as required formlr3
’sbenchmark()
function), instead ofbenchmark()
callbatchmark()
which populates anExperimentRegistry
for the computational jobs of the benchmark. You are now in the world ofbatchtools
where you can selectively submit jobs with different resources, monitor the progress or resubmit as needed. - After the computations are finished, collect the results with
reduceResultsBatchmark()
to return tomlr3
. The resulting object is a regularBenchmarkResult
.
Example
library("mlr3")
library("batchtools")
library("mlr3batchmark")
tasks = tsks(c("iris", "sonar"))
learners = lrns(c("classif.featureless", "classif.rpart"))
resamplings = rsmp("cv", folds = 3)
design = benchmark_grid(
tasks = tasks,
learners = learners,
resamplings = resamplings
)
reg = makeExperimentRegistry(NA)
## No readable configuration file found
## Created registry in '/tmp/Rtmp8DlMZQ/registry704553adf7a88' using cluster functions 'Interactive'
ids = batchmark(design, reg = reg)
## Adding algorithm 'run_learner'
## Adding problem 'b39ef23a66b1f1ee'
## Exporting new objects: '5ec484de3f93431b' ...
## Exporting new objects: '7c35d835f3dfae37' ...
## Exporting new objects: '70dd22724e5c724d' ...
## Adding 6 experiments ('b39ef23a66b1f1ee'[1] x 'run_learner'[2] x repls[3]) ...
## Adding problem '76c4fc7a533d41b7'
## Exporting new objects: 'b209de197d6cbe75' ...
## Adding 6 experiments ('76c4fc7a533d41b7'[1] x 'run_learner'[2] x repls[3]) ...
submitJobs()
## Submitting 12 jobs in 12 chunks using cluster functions 'Interactive' ...
getStatus()
## Status for 12 jobs at 2023-11-13 19:32:20:
## Submitted : 12 (100.0%)
## -- Queued : 0 ( 0.0%)
## -- Started : 12 (100.0%)
## ---- Running : 0 ( 0.0%)
## ---- Done : 12 (100.0%)
## ---- Error : 0 ( 0.0%)
## ---- Expired : 0 ( 0.0%)
reduceResultsBatchmark()
## <BenchmarkResult> of 12 rows with 4 resampling runs
## nr task_id learner_id resampling_id iters warnings errors
## 1 iris classif.featureless cv 3 0 0
## 2 iris classif.rpart cv 3 0 0
## 3 sonar classif.featureless cv 3 0 0
## 4 sonar classif.rpart cv 3 0 0
Resources
- The Large-Scale Benchmarking chapter of the mlr3 book.