Analyzing and Visualizing Symptom Checker Performance.
symptomcheckR package
Easy analysis and visualization of symptom checker performance metrics
The symptomcheckR package can be used to analyze the performance of symptom checkers across various metrics. Since many studies report different metrics, we aimed to standardize performance reporting by developing more reliable metrics that future studies should include. They can be found in Kopka et al. (2023). To make it easier for researchers and other stakeholders to report these metrics and compare different symptom checkers, we developed this package.
Installation
install.packages("symptomcheckR")
Usage
First, load the package:
library(symptomcheckR)
Now you can load the data set to test the commands:
data(symptomcheckRdata)
This will load the data set. You can run all commands from this package now. These include the following metrics:
Accuracy:
accuracy_value <- get_accuracy(
symptomcheckRdata,
correct = "Correct_Triage_Advice_provided_from_app",
apps = "App_name")
plot_accuracy(accuracy_value)
Accuracy by triage level:
get_accuracy_by_triage(
symptomcheckRdata,
correct = "Correct_Triage_Advice_provided_from_app",
triagelevel = "Goldstandard_solution",
apps = "App_name")
plot_accuracy_by_triage(accuracy_value_by_triage)
Safety of advice:
safety <- get_safety_of_advice(
data = symptomcheckRdata,
triagelevel_correct = "Goldstandard_solution",
triagelevel_advice = "Triage_advice_from_app",
order_triagelevel = c("Emergency", "Non-Emergency", "Self-care"),
apps = "App_name")
plot_safety_of_advice(safety)
Comprehensiveness:
comprehensiveness <- get_comprehensiveness(
data = symptomcheckRdata,
triagelevel_advice = "Triage_advice_from_app",
vector_not_entered = c(NA),
apps = "App_name")
plot_comprehensiveness(comprehensiveness)
Inclination to overtriage:
inclination_to_overtriage <- get_inclination_overtriage(
data = symptomcheckRdata,
triagelevel_correct = "Goldstandard_solution",
triagelevel_advice = "Triage_advice_from_app",
order_triagelevel = c("Emergency", "Non-Emergency", "Self-care"),
apps = "App_name")
plot_inclination_overtriage(inclination_to_overtriage)
Item Difficulty of each vignette:
get_item_difficulty(
data = symptomcheckRdata,
correct = "Correct_Triage_Advice_provided_from_app",
vignettes = "Vignette_id")
For all of these commands, you can set CI to TRUE (by passing the argument CI == TRUE to the corresponding function) to obtain 95% confidence intervals.
Capability Comparison Score:
ccs <- get_ccs(
data = symptomcheckRdata,
correct = "Correct_Triage_Advice_provided_from_app",
vignettes = "Vignette_id",
apps = "App_name")
plot_ccs(ccs)
Capability Comparison Score on each triage level:
get_ccs_by_triage <- get_ccs_by_triage(
data = symptomcheckRdata,
correct = "Correct_Triage_Advice_provided_from_app",
vignettes = "Vignette_id",
apps = "App_name",
triage = "Goldstandard_solution")
Users can also get a performance overview for one or multiple symptom checkers. To get the overview (and publication-ready figures) for a single symptom checker, you can use the following command:
df_individual <- symptomcheckRdata %>%
filter(App_name == "Ask NHS")
plot_performance_single(
data = df_individual,
triagelevel_correct = "Goldstandard_solution",
triagelevel_advice = "Triage_advice_from_app",
order_triagelevel = c("Emergency", "Non-Emergency", "Self-care"),
vector_not_entered = c(NA))
To compare multiple symptom checkers (and produce publication-ready figures), you can use the following command:
plot_performance_multiple(
data = symptomcheckRdata,
triagelevel_correct = "Goldstandard_solution",
triagelevel_advice = "Triage_advice_from_app",
order_triagelevel = c("Emergency", "Non-Emergency", "Self-care"),
vector_not_entered = c(NA),
vignettes = "Vignette_id",
apps = "App_name")
To calculate the inter-rater reliability (if multiple inputters were involved), you can use the following command:
get_irr(
data = df,
ratings = c("datarater1", "datarater2", "datarater3"),
order_triagelevel = c("Emergency", "Non-Emergency", "Self-care"),
)