MyNixOS website logo
Description

Additional Functions for Model Tuning.

The ability to tune models is important. 'finetune' enhances the 'tune' package by providing more specialized methods for finding reasonable values of model tuning parameters. Two racing methods described by Kuhn (2014) <arXiv:1405.6974> are included. An iterative search method using generalized simulated annealing (Bohachevsky, Johnson and Stein, 1986) <doi:10.1080/00401706.1986.10488128> is also included.

finetune

R-CMD-check Codecov testcoverage Lifecycle

finetune contains some extra functions for model tuning that extend what is currently in the tune package. You can install the CRAN version of the package with the following code:

install.packages("finetune")

To install the development version of the package, run:

# install.packages("pak")
pak::pak("tidymodels/finetune")

There are two main sets of tools in the package: simulated annealing and racing.

Tuning via simulated annealing optimization is an iterative search tool for finding good values:

library(tidymodels)
library(finetune)

# Syntax very similar to `tune_grid()` or `tune_bayes()`: 

## -----------------------------------------------------------------------------

data(two_class_dat, package = "modeldata")

set.seed(1)
rs <- bootstraps(two_class_dat, times = 10) # more resamples usually needed

# Optimize a regularized discriminant analysis model
library(discrim)
rda_spec <-
  discrim_regularized(frac_common_cov = tune(), frac_identity = tune()) %>%
  set_engine("klaR")

## -----------------------------------------------------------------------------

set.seed(2)
sa_res <- 
  rda_spec %>% 
  tune_sim_anneal(Class ~ ., resamples = rs, iter = 20, initial = 4)
#> Optimizing roc_auc
#> Initial best: 0.86480
#> 1 ♥ new best           roc_auc=0.87327 (+/-0.004592)
#> 2 ♥ new best           roc_auc=0.87915 (+/-0.003864)
#> 3 ◯ accept suboptimal  roc_auc=0.87029 (+/-0.004994)
#> 4 + better suboptimal  roc_auc=0.87171 (+/-0.004717)
#> 5 ◯ accept suboptimal  roc_auc=0.86944 (+/-0.005081)
#> 6 ◯ accept suboptimal  roc_auc=0.86812 (+/-0.0052)
#> 7 ♥ new best           roc_auc=0.88172 (+/-0.003647)
#> 8 ◯ accept suboptimal  roc_auc=0.87678 (+/-0.004276)
#> 9 ◯ accept suboptimal  roc_auc=0.8627 (+/-0.005784)
#> 10 + better suboptimal  roc_auc=0.87003 (+/-0.005106)
#> 11 + better suboptimal  roc_auc=0.87088 (+/-0.004962)
#> 12 ◯ accept suboptimal  roc_auc=0.86803 (+/-0.005195)
#> 13 ◯ accept suboptimal  roc_auc=0.85294 (+/-0.006498)
#> 14 ─ discard suboptimal roc_auc=0.84689 (+/-0.006867)
#> 15 ✖ restart from best  roc_auc=0.85021 (+/-0.006623)
#> 16 ◯ accept suboptimal  roc_auc=0.87607 (+/-0.004318)
#> 17 ◯ accept suboptimal  roc_auc=0.87245 (+/-0.004799)
#> 18 + better suboptimal  roc_auc=0.87706 (+/-0.004131)
#> 19 ◯ accept suboptimal  roc_auc=0.87213 (+/-0.004791)
#> 20 ◯ accept suboptimal  roc_auc=0.86218 (+/-0.005773)
show_best(sa_res, metric = "roc_auc", n = 2)
#> # A tibble: 2 × 9
#>   frac_common_cov frac_identity .metric .estimator  mean     n std_err .config
#>             <dbl>         <dbl> <chr>   <chr>      <dbl> <int>   <dbl> <chr>  
#> 1           0.308        0.0166 roc_auc binary     0.882    10 0.00365 Iter7  
#> 2           0.121        0.0474 roc_auc binary     0.879    10 0.00386 Iter2  
#> # ℹ 1 more variable: .iter <int>

The second set of methods are for racing. We start off by doing a small set of resamples for all of the grid points, then statistically testing to see which ones should be dropped or investigated more. The two methods here are based on those should in Kuhn (2014).

For example, using an ANOVA-type analysis to filter out parameter combinations:

set.seed(3)
grid <-
  rda_spec %>%
  extract_parameter_set_dials() %>%
  grid_max_entropy(size = 20)

ctrl <- control_race(verbose_elim = TRUE)

set.seed(4)
grid_anova <- 
  rda_spec %>% 
  tune_race_anova(Class ~ ., resamples = rs, grid = grid, control = ctrl)
#> ℹ Evaluating against the initial 3 burn-in resamples.
#> ℹ Racing will maximize the roc_auc metric.
#> ℹ Resamples are analyzed in a random order.
#> ℹ Bootstrap10: 14 eliminated; 6 candidates remain.
#> 
#> ℹ Bootstrap04: 2 eliminated; 4 candidates remain.
#> 
#> ℹ Bootstrap03: All but one parameter combination were eliminated.

show_best(grid_anova, metric = "roc_auc", n = 2)
#> # A tibble: 1 × 8
#>   frac_common_cov frac_identity .metric .estimator  mean     n std_err .config  
#>             <dbl>         <dbl> <chr>   <chr>      <dbl> <int>   <dbl> <chr>    
#> 1           0.831        0.0207 roc_auc binary     0.881    10 0.00386 Preproce…

tune_race_win_loss() can also be used. It treats the tuning parameters as sports teams in a tournament and computed win/loss statistics.

set.seed(4)
grid_win_loss<- 
  rda_spec %>% 
  tune_race_win_loss(Class ~ ., resamples = rs, grid = grid, control = ctrl)
#> ℹ Racing will maximize the roc_auc metric.
#> ℹ Resamples are analyzed in a random order.
#> ℹ Bootstrap10: 3 eliminated; 17 candidates remain.
#> 
#> ℹ Bootstrap04: 2 eliminated; 15 candidates remain.
#> 
#> ℹ Bootstrap03: 2 eliminated; 13 candidates remain.
#> 
#> ℹ Bootstrap01: 1 eliminated; 12 candidates remain.
#> 
#> ℹ Bootstrap07: 1 eliminated; 11 candidates remain.
#> 
#> ℹ Bootstrap05: 1 eliminated; 10 candidates remain.
#> 
#> ℹ Bootstrap08: 1 eliminated; 9 candidates remain.

show_best(grid_win_loss, metric = "roc_auc", n = 2)
#> # A tibble: 2 × 8
#>   frac_common_cov frac_identity .metric .estimator  mean     n std_err .config  
#>             <dbl>         <dbl> <chr>   <chr>      <dbl> <int>   <dbl> <chr>    
#> 1           0.831        0.0207 roc_auc binary     0.881    10 0.00386 Preproce…
#> 2           0.119        0.0470 roc_auc binary     0.879    10 0.00387 Preproce…

Contributing

This project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Metadata

Version

1.2.0

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows