MyNixOS website logo
Description

Testing Workbench for Precision-Recall Curves.

A testing workbench to evaluate tools that calculate precision-recall curves. Saito and Rehmsmeier (2015) <doi:10.1371/journal.pone.0118432>.

prcbench

R-CMD-check AppVeyor BuildStatus codecov.io CodeFactor CRAN_Status_Badge CRAN_Logs_Badge

The aim of the prcbench package is to provide a testing workbench for evaluating precision-recall curves under various conditions. It contains integrated interfaces for the following five tools. It also contains predefined test data sets.

ToolLanguageLink
precrecRTool web site, CRAN
ROCRRTool web site, CRAN
PRROCRCRAN
AUCCalculatorJavaTool web site
PerfMeasRCRAN

Disclaimer: prcbench was originally develop to help our precrec library in order to provide fast and accurate calculations of precision-recall curves with extra functionality.

Accuracy evaluation of precision-recall curves

prcbench uses pre-defined test sets to help evaluate the accuracy of precision-recall curves.

  1. create_toolset: creates objects of different tools for testing (5 different tools)
  2. create_testset: selects pre-defined data sets (c1, c2, and c3)
  3. run_evalcurve: evaluates the selected tools on the simulation data
  4. autoplot: shows the results with ggplot2 and patchwork
## Load library
library(prcbench)

## Plot base points and the result of 5 tools on pre-defined test sets (c1, c2, and c3)
toolset <- create_toolset(c("precrec", "ROCR", "AUCCalculator", "PerfMeas", "PRROC"))
testset <- create_testset("curve", c("c1", "c2", "c3"))
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1, ncol = 3, nrow = 2)

Running-time evaluation of precision-recall curves

prcbench helps create simulation data to measure computational times of creating precision-recall curves.

  1. create_toolset: creates objects of different tools for testing
  2. create_testset: creates simulation data
  3. run_benchmark: evaluates the selected tools on the simulation data
## Load library
library(prcbench)

## Run benchmark for auc5 (5 tools) on b10 (balanced 5 positives and 5 negatives)
toolset <- create_toolset(set_names = "auc5")
testset <- create_testset("bench", "b10")
res <- run_benchmark(testset, toolset)

print(res)
testsettoolsettoolnameminlqmeanmedianuqmaxneval
b10auc5AUCCalculator0.930.961.121.001.001.685
b10auc5PerfMeas0.060.060.080.060.070.175
b10auc5precrec3.403.453.733.473.584.745
b10auc5PRROC0.140.140.170.140.160.285
b10auc5ROCR1.571.591.691.601.632.065

Documentation

  • Introduction to prcbench – a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "prcbench") in R.

  • Help pages – all the functions including the S3 generics have their own help pages with plenty of examples. View the main help page with help(package = "prcbench") in R.

Installation

CRAN

install.packages("prcbench")

Dependencies

AUCCalculator requires a Java runtime environment (>= 6) if AUCCalculator needs to be evaluated.

GitHub

You can install a development version of prcbench from our GitHub repository.

devtools::install_github("evalclass/prcbench")
  1. Make sure you have a working development environment.

    • Windows: Install Rtools (available on the CRAN website).

    • Mac: Install Xcode from the Mac App Store.

    • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).

  2. Install devtools from CRAN with install.packages("devtools").

  3. Install prcbench from the GitHub repository with devtools::install_github("evalclass/prcbench").

Troubleshooting

microbenchmark

microbenchmark does not work on some OSs. prcbench uses system.time when microbenchmark is not available.

rJava

  • Some OSs require en extra configuration step after rJava installation.
sudo R CMD javareconf
  • JDKs
  1. Oracle JDK
  2. OpenJDK
  • JDKs for macOS
  1. AdoptOpenJDK
  2. AdoptOpenJDK with homebrew
install.packages("rJava", configure.args = "--disable-jri")

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links

Metadata

Version

1.1.8

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows