Description
Performance Assessment of Binary Classifier with Visualization.
Description
Sensitivity (or recall or true positive rate), false positive rate, specificity, precision (or positive predictive value), negative predictive value, misclassification rate, accuracy, F-score- these are popular metrics for assessing performance of binary classifier for certain threshold. These metrics are calculated at certain threshold values. Receiver operating characteristic (ROC) curve is a common tool for assessing overall diagnostic ability of the binary classifier. Unlike depending on a certain threshold, area under ROC curve (also known as AUC), is a summary statistic about how well a binary classifier performs overall for the classification task. ROCit package provides flexibility to easily evaluate threshold-bound metrics. Also, ROC curve, along with AUC, can be obtained using different methods, such as empirical, binormal and non-parametric. ROCit encompasses a wide variety of methods for constructing confidence interval of ROC curve and AUC. ROCit also features the option of constructing empirical gains table, which is a handy tool for direct marketing. The package offers options for commonly used visualization, such as, ROC curve, KS plot, lift plot. Along with in-built default graphics setting, there are rooms for manual tweak by providing the necessary values as function arguments. ROCit is a powerful tool offering a range of things, yet it is very easy to use.
README.md
ROCit 2.1.2
Author
Md Riaz Ahmed Khan [email protected] Thomas Brandenburger, PhD [email protected]
Maintainer
Md Riaz Ahmed Khan [email protected]
ROCit
The goal of ROCit is to easily compute, summarize, and visualize the performance of a binary classifier. ROCit package provides flexibility to easily evaluate threshold-bound metrics, such as, accuracy, misclassification rate, sensitivity, specificity, F-Score. . Also, ROC curve, along with AUC, can be obtained using different methods, such as empirical, binormal and non-parametric. ROCit encompasses a wide variety of methods for constructing confidence interval of ROC curve and AUC. ROCit features the option of constructing empirical gains table, which is a handy tool for direct marketing. The package offers options for commonly used visualization, such as, ROC curve, KS plot, lift plot.
Example
This is a basic example which shows the workflow of assessing a binary classifier.
library(ROCit)
data("Loan")
summary(Loan$Status)
#> CO FP
#> 131 769
Make a response
- Label 1 for charged off, 0 for fully paid
class <- ifelse(Loan$Status == "FP", 0, 1)
Use score as a predictor of defaulting
score <- Loan$Score
Calculate threshold-bound metrics
m<- measureit(score = score, class = class,
measure = c("ACC", "SENS", "SPEC", "FSCR"))
mymetrics <- as.data.frame(cbind(Cutoff = m$Cutoff, Depth = m$Depth,
Accuracy = m$ACC, Sensitivity = m$SENS,
Specificity = m$SPEC, `F-Score` = m$FSCR))
head(mymetrics)
#> Cutoff Depth Accuracy Sensitivity Specificity F-Score
#> 1 Inf 0.000000000 0.8544444 0.000000000 1.0000000 NaN
#> 2 269.1711 0.001111111 0.8555556 0.007633588 1.0000000 0.01515152
#> 3 267.3342 0.002222222 0.8544444 0.007633588 0.9986996 0.01503759
#> 4 266.5938 0.003333333 0.8533333 0.007633588 0.9973992 0.01492537
#> 5 265.2553 0.004444444 0.8522222 0.007633588 0.9960988 0.01481481
#> 6 263.8303 0.005555556 0.8533333 0.015267176 0.9960988 0.02941176
tail(mymetrics)
#> Cutoff Depth Accuracy Sensitivity Specificity F-Score
#> 897 103.236891 0.9955556 0.1500000 1 0.00520156 0.2551120
#> 898 93.941202 0.9966667 0.1488889 1 0.00390117 0.2548638
#> 899 70.836943 0.9977778 0.1477778 1 0.00260078 0.2546161
#> 900 -2.673036 0.9988889 0.1466667 1 0.00130039 0.2543689
#> 901 -5.449033 1.0000000 0.1455556 1 0.00000000 0.2541222
#> 902 -Inf 1.0000000 0.1455556 1 0.00000000 0.2541222
ROC curve
rocit_object_empirical <- rocit(score = score, class = class)
summary(rocit_object_empirical)
#>
#> Empirical ROC curve
#> Number of postive responses : 131
#> Number of negative responses : 769
#> Area under curve : 0.666315925311945
Plot ROC curve
# plot(rocit_object_empirical, YIndex = F, values = F)
Confidence interval of AUC
ci.auc <- ciAUC(rocit_object_empirical, level = 0.95)
print(ci.auc)
#>
#> estimated AUC : 0.666315925311945
#> AUC estimation method : empirical
#>
#> CI of AUC
#> confidence level = 95%
#> lower = 0.612575921206968 upper = 0.720055929416921
Confidence interval of ROC curve
ci.roc <- ciROC(rocit_object_empirical, level = 0.9)
# plot(ci.roc)
Gains table
gainstable <- gainstable(rocit_object_empirical, ngroup = 8)
print(gainstable, maxdigit = 2)
#> Bucket Obs CObs Depth Resp CResp RespRate CRespRate CCapRate Lift CLift
#> 1 1 112 112 0.12 30 30 0.27 0.27 0.23 1.84 1.84
#> 2 2 113 225 0.25 25 55 0.22 0.24 0.42 1.52 1.68
#> 3 3 113 338 0.38 21 76 0.19 0.22 0.58 1.28 1.54
#> 4 4 112 450 0.50 18 94 0.16 0.21 0.72 1.10 1.44
#> 5 5 112 562 0.62 10 104 0.09 0.19 0.79 0.61 1.27
#> 6 6 113 675 0.75 16 120 0.14 0.18 0.92 0.97 1.22
#> 7 7 113 788 0.88 5 125 0.04 0.16 0.95 0.30 1.09
#> 8 8 112 900 1.00 6 131 0.05 0.15 1.00 0.37 1.00
# plot(gainstable)