Why does tune_race_anova() fail at a task, that tune_grid() is able to handle?

Question 1

The following example illustrates the parameter tuning for the cells dataset using the tune package. However, if I try to switch from the tune packages grid() to finetunes tune_race_anova(), I am getting the following error message: "Error: There were no valid metrics for the ANOVA model."

With my reprex you can try both methods:

suppressPackageStartupMessages(library(tidymodels))
suppressPackageStartupMessages(library(doParallel))
suppressPackageStartupMessages(library(recipeselectors))
suppressPackageStartupMessages(library(finetune))

data(cells, package = "modeldata")

cells <- cells %>% select(-case)
set.seed(31)
split <- initial_split(cells, prop = 0.8)
train <- training(split)
test <- testing(split)

rec <-
    recipe(class ~ ., data = train) %>%
    step_corr(all_predictors(), threshold = 0.9) %>% 
    step_select_roc(all_predictors(), outcome = "class", top_p = tune())

# xgboost model
xgb_spec <- boost_tree(
    trees = tune(), 
    tree_depth = tune(), min_n = tune(), 
    loss_reduction = tune(),                    
    sample_size = tune(), mtry = tune(),         
    learn_rate = tune(),                        
    stop_iter = tune()
) %>% 
    set_engine("xgboost") %>% 
    set_mode("classification")

# grids
xgb_grid <- grid_latin_hypercube(
    trees(),
    tree_depth(),
    min_n(),
    loss_reduction(),
    sample_size = sample_prop(),
    finalize(mtry(), train),
    learn_rate(),
    stop_iter(range = c(5L,50L)),
    size = 5
)
rec_grid <- grid_latin_hypercube(
    parameters(rec) %>% 
        update(top_p = top_p(c(0,30))) ,
    size = 5
)
comp_grid <- merge(xgb_grid, rec_grid)

model_metrics <- metric_set(roc_auc)  

rs <- vfold_cv(cells)

ctrl <- control_race(pkgs = "recipeselectors")
#ctrl <- control_grid(pkgs = "recipeselectors")

# tune
cores <- parallel::detectCores(logical = FALSE)
cl <- makePSOCKcluster(cores)
registerDoParallel(cl)
set.seed(234)
rfe_res <-
    xgb_spec %>% 
    tune_race_anova(
    #tune_grid(
        preprocessor = rec,
        resamples = rs,
        grid = comp_grid,
        control = ctrl
    )
stopCluster(cl)

Why does tune_race_anova() fail at a task, that tune_grid() is able to handle?

Question 2

I think your problem in this particular case may be that your metric is too close to the same for every possible model. You can see this with a simpler example, but it looks like it is happening for when you add in more steps too:

library(tidymodels)
library(tune)
#> Registered S3 method overwritten by 'finetune':
#>   method            from
#>   obj_sum.tune_race tune
data(cells, package = "modeldata")

set.seed(31)
split <- cells %>% 
  select(-case) %>%
  initial_split(prop = 0.8)

set.seed(234)
folds <- training(split) %>% vfold_cv(v = 3)
folds
#> #  3-fold cross-validation 
#> # A tibble: 3 × 2
#>   splits             id   
#>    
#> 1  Fold1
#> 2  Fold2
#> 3  Fold3

xgb_spec <- boost_tree(mode = "classification", trees = tune()) 

set.seed(234)
workflow(class ~ ., xgb_spec) %>% 
  grid(
    resamples = folds,
    grid = 5
  ) %>%
  collect_metrics()
#> # A tibble: 10 × 7
#>    trees .metric  .estimator  mean     n std_err .config             
#>                         
#>  1   354 accuracy binary     0.834     3 0.00619 Preprocessor1_Model1
#>  2   354 roc_auc  binary     0.907     3 0.00340 Preprocessor1_Model1
#>  3   736 accuracy binary     0.836     3 0.00619 Preprocessor1_Model2
#>  4   736 roc_auc  binary     0.907     3 0.00314 Preprocessor1_Model2
#>  5   972 accuracy binary     0.835     3 0.00638 Preprocessor1_Model3
#>  6   972 roc_auc  binary     0.907     3 0.00316 Preprocessor1_Model3
#>  7  1396 accuracy binary     0.836     3 0.00619 Preprocessor1_Model4
#>  8  1396 roc_auc  binary     0.907     3 0.00302 Preprocessor1_Model4
#>  9  1949 accuracy binary     0.835     3 0.00577 Preprocessor1_Model5
#> 10  1949 roc_auc  binary     0.907     3 0.00292 Preprocessor1_Model5

^{Created on 2022-02-22 by the reprex package (v2.0.1)}

I think finetune may not be handling the metrics being the same well, and is erroring in a confusing way.

Answer 1

I think your problem in this particular case may be that your metric is too close to the same for every possible model. You can see this with a simpler example, but it looks like it is happening for when you add in more steps too:

library(tidymodels)
library(tune)
#> Registered S3 method overwritten by 'finetune':
#>   method            from
#>   obj_sum.tune_race tune
data(cells, package = "modeldata")

set.seed(31)
split <- cells %>% 
  select(-case) %>%
  initial_split(prop = 0.8)

set.seed(234)
folds <- training(split) %>% vfold_cv(v = 3)
folds
#> #  3-fold cross-validation 
#> # A tibble: 3 × 2
#>   splits             id   
#>    
#> 1  Fold1
#> 2  Fold2
#> 3  Fold3

xgb_spec <- boost_tree(mode = "classification", trees = tune()) 

set.seed(234)
workflow(class ~ ., xgb_spec) %>% 
  grid(
    resamples = folds,
    grid = 5
  ) %>%
  collect_metrics()
#> # A tibble: 10 × 7
#>    trees .metric  .estimator  mean     n std_err .config             
#>                         
#>  1   354 accuracy binary     0.834     3 0.00619 Preprocessor1_Model1
#>  2   354 roc_auc  binary     0.907     3 0.00340 Preprocessor1_Model1
#>  3   736 accuracy binary     0.836     3 0.00619 Preprocessor1_Model2
#>  4   736 roc_auc  binary     0.907     3 0.00314 Preprocessor1_Model2
#>  5   972 accuracy binary     0.835     3 0.00638 Preprocessor1_Model3
#>  6   972 roc_auc  binary     0.907     3 0.00316 Preprocessor1_Model3
#>  7  1396 accuracy binary     0.836     3 0.00619 Preprocessor1_Model4
#>  8  1396 roc_auc  binary     0.907     3 0.00302 Preprocessor1_Model4
#>  9  1949 accuracy binary     0.835     3 0.00577 Preprocessor1_Model5
#> 10  1949 roc_auc  binary     0.907     3 0.00292 Preprocessor1_Model5

^{Created on 2022-02-22 by the reprex package (v2.0.1)}

I think finetune may not be handling the metrics being the same well, and is erroring in a confusing way.

Why does tune_race_anova() fail at a task, that tune_grid() is able to handle?

Why does tune_race_anova() fail at a task, that tune_grid() is able to handle?

0 Comment

0 Comment

About the Author

Question Info

Related Question

我需要批量删除子文件夹和iTunes在我的音乐文件夹中创建的内容

Why does windows command prompt keep showing python 3.9.12 even though I've uninstalled all versions of python?

BeautifulSoup to find a HTML tag that contains tags with specific class

如何通过一个中间函数传递Formik表单值到handleSubmit事件?

UnhandledException的堆栈跟踪

Is there anything in Jinja that would allow me to select only the data from an array without the `['']`

flutter permission_handler构建失败

需要Win32应用程序在微软Intune

HttpPlatformHandler已被ASP弃用.iis10的NET Core对应的配置是什么?

Realloc() causes leaks that I cannot find