Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation.

pairwiseLLM: Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation

pairwiseLLM provides a unified, extensible framework for generating, submitting, and modeling pairwise comparisons of writing quality using large language models (LLMs).

It includes:

• Unified live and batch APIs across OpenAI, Anthropic, and Gemini
• A prompt template registry with tested templates designed to reduce positional bias
• Positional-bias diagnostics (forward vs reverse design)
• Bradley–Terry (BT) and Elo modeling
• Consistent data structures for all providers

Vignettes

Several vignettes are available to demonstrate functionality.

For basic function usage, see:

• vignette("getting-started")

For advanced batch processing workflows, see:

• vignette("advanced-batch-workflows")

For information on prompt evaluation and positional-bias diagnostics, see:

• vignette("prompt-template-bias")

Supported Models

The following models are confirmed to work for pairwise comparisons:

₁ via the together.ai API

₂ via Ollama on a local machine

Batch APIs are currently available for OpenAI, Anthropic, and Gemini only. Models accessed via Together.ai and Ollama are supported for live comparisons via submit_llm_pairs() / llm_compare_pair().

Installation

Once the package is available on CRAN, install with:

install.packages("pairwiseLLM")

To install the development version from GitHub:

# install.packages("pak")
pak::pak("shmercer/pairwiseLLM")

Load the package:

library(pairwiseLLM)

Core Concepts

At a high level, pairwiseLLM workflows follow this structure:

1. Writing samples – e.g., essays, constructed responses, short answers.
2. Trait – a rating dimension such as “overall quality” or “organization”.
3. Pairs – pairs of samples to be compared for that trait.
4. Prompt template – instructions + placeholders for {TRAIT_NAME}, {TRAIT_DESCRIPTION}, {SAMPLE_1}, {SAMPLE_2}.
5. Backend – which provider/model to use (OpenAI, Anthropic, Gemini, Together, Ollama).
6. Modeling – convert pairwise results to latent scores via BT or Elo.

The package provides helpers for each step.

Live Comparisons

Use the unified API:

• llm_compare_pair() — compare one pair
• submit_llm_pairs() — compare many pairs at once

Example:

data("example_writing_samples")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(5, seed = 123) |>
  randomize_pair_order()

td <- trait_description("overall_quality")
tmpl <- get_prompt_template("default")

res <- submit_llm_pairs(
  pairs             = pairs,
  backend           = "openai",
  model             = "gpt-4o",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl
)

Batch Comparisons

Large-scale runs use:

• llm_submit_pairs_batch()
• llm_download_batch_results()

Example:

batch <- llm_submit_pairs_batch(
  backend           = "anthropic",
  model             = "claude-sonnet-4-5",
  pairs             = pairs,
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl
)

results <- llm_download_batch_results(batch)

API Keys

pairwiseLLM reads keys only from environment variables.
Keys are never printed, never stored, and never written to disk.

You can verify which providers are available using:

check_llm_api_keys()

This returns a tibble showing whether R can see the required keys for:

• OpenAI
• Anthropic
• Google Gemini
• Together.ai

Setting API Keys

You may set keys temporarily for the current R session:

Sys.setenv(OPENAI_API_KEY = "your-key-here")
Sys.setenv(ANTHROPIC_API_KEY = "your-key-here")
Sys.setenv(GEMINI_API_KEY = "your-key-here")
Sys.setenv(TOGETHER_API_KEY = "your-key-here")

…but for normal use and for reproducible analyses, it is strongly recommended
to store them in your ~/.Renviron file.

Recommended method: Adding keys to ~/.Renviron

Open your .Renviron file:

usethis::edit_r_environ()

Add the following lines:

OPENAI_API_KEY="your-openai-key"
ANTHROPIC_API_KEY="your-anthropic-key"
GEMINI_API_KEY="your-gemini-key"
TOGETHER_API_KEY="your-together-key"

Save the file, then restart R.

You can confirm that R now sees the keys:

check_llm_api_keys()

Prompt Templates & Registry

pairwiseLLM includes:

• A default template tested for positional bias
• Support for multiple templates stored by name
• User-defined templates via register_prompt_template()

View available templates

list_prompt_templates()
#> [1] "default" "test1"   "test2"   "test3"   "test4"   "test5"

Show the default template (truncated)

tmpl <- get_prompt_template("default")
cat(substr(tmpl, 1, 400), "...\n")
#> You are a debate adjudicator. Your task is to weigh the comparative strengths of two writing samples regarding a specific trait.
#> 
#> TRAIT: {TRAIT_NAME}
#> DEFINITION: {TRAIT_DESCRIPTION}
#> 
#> SAMPLES:
#> 
#> === SAMPLE_1 ===
#> {SAMPLE_1}
#> 
#> === SAMPLE_2 ===
#> {SAMPLE_2}
#> 
#> EVALUATION PROCESS (Mental Simulation):
#> 
#> 1.  **Advocate for SAMPLE_1**: Mentally list the single strongest point of evidence that makes SAMPLE_1 the  ...

Register your own template

register_prompt_template("my_template", "
Compare two essays for {TRAIT_NAME}…

{TRAIT_NAME} is defined as {TRAIT_DESCRIPTION}.

SAMPLE 1:
{SAMPLE_1}

SAMPLE 2:
{SAMPLE_2}

<BETTER_SAMPLE>SAMPLE_1</BETTER_SAMPLE> or
<BETTER_SAMPLE>SAMPLE_2</BETTER_SAMPLE>
")

Use it in a submission:

tmpl <- get_prompt_template("my_template")

Trait Descriptions

Traits define what “quality” means.

trait_description("overall_quality")
#> $name
#> [1] "Overall Quality"
#> 
#> $description
#> [1] "Overall quality of the writing, considering how well ideas are expressed,\n      how clearly the writing is organized, and how effective the language and\n      conventions are."

You can also provide custom traits:

trait_description(
  custom_name        = "Clarity",
  custom_description = "How understandable, coherent, and well structured the ideas are."
)

Positional Bias Testing

LLMs often show a first-position or second-position bias.
pairwiseLLM includes explicit tools for testing this.

Typical workflow

pairs_fwd <- make_pairs(example_writing_samples)
pairs_rev <- sample_reverse_pairs(pairs_fwd, reverse_pct = 1.0)

Submit:

res_fwd <- submit_llm_pairs(pairs_fwd, model = "gpt-4o", backend = "openai", ...)
res_rev <- submit_llm_pairs(pairs_rev, model = "gpt-4o", backend = "openai", ...)

Compute bias:

cons <- compute_reverse_consistency(res_fwd, res_rev)
bias <- check_positional_bias(cons)

cons$summary
bias$summary

Positional-bias tested templates

Five included templates have been tested across different backend providers. Complete details are presented in a vignette: vignette("prompt-template-bias")

Bradley–Terry & Elo Modeling

Bradley–Terry (BT)

bt_data <- build_bt_data(res)
bt_fit <- fit_bt_model(bt_data)
summarize_bt_fit(bt_fit)

Elo Modeling

# res: output from submit_llm_pairs() / llm_submit_pairs_batch()
elo_data <- build_elo_data(res)
elo_fit  <- fit_elo_model(elo_data, runs = 5)

elo_fit$elo
elo_fit$reliability
elo_fit$reliability_weighted

Live vs Batch Summary

Contributing

Contributions to pairwiseLLM are very welcome!

• Bug reports (with reproducible examples when possible)
• Feature requests, ideas, and discussion
• Pull requests improving:
  • functionality
  • documentation
  • examples / vignettes
  • test coverage
• Backend integrations (e.g., additional LLM providers or local inference engines)
• Modeling extensions

Reporting issues

If you encounter a problem:

1. Run:

   devtools::session_info()
   

2. Include:

   • reproducible code
   • the error message
   • the model/backend involved
   • your operating system

3. Open an issue at:
   https://github.com/shmercer/pairwiseLLM/issues

License

MIT License. See LICENSE.

Package Author and Maintainer

• Sterett H. Mercer – University of British Columbia
  UBC Faculty Profile: ResearchGate: <https://www.researchgate.net/profile/Sterett_Mercer/
  Google Scholar: https://scholar.google.ca/citations?user=YJg4svsAAAAJ&hl=en

Citation

Mercer, S. H. (2025). pairwiseLLM: Pairwise writing quality comparisons with large language models (Version 1.0.0) [R package; Computer software]. https://github.com/shmercer/pairwiseLLM.