A Comprehensive Hit or Miss Probabilistic Entity Resolution Model.

chomper

Overview

chomper is an R package that provides a Comprehensive Hit Or Miss Entity Resolution (CHOMPER) models.

Key Features

• Multiple Inference Approaches: Implements three inference methods:

  • MCMC: Markov Chain Monte Carlo with split and merge process
  • EVIL: Evolutionary Variational Inference for record Linkage
  • CAVI: Single Coordinate Ascent Variational Inference

• Locally Varying Hit Mechanism: Accounts for the attributes with multiple truths

• Flexible Data Support: Handles both discrete and continuous fields

• Parallel Computing: Multi-threading support for faster EVIL estimation

Installation

You can install chomper from GitHub with:

# install.packages("devtools")
devtools::install_github("hjkim8987/chomper", dependencies = TRUE, build_vignettes = TRUE)

Quick Start

library(chomper)

# Generate sample data for testing
sample_data <- generate_sample_data(
  n_entities = 100,
  n_files = 3,
  overlap_ratio = 0.7,
  discrete_columns = c(1, 2),
  discrete_levels = c(5, 5),
  continuous_columns = c(3, 4),
  continuous_params = matrix(c(0, 0, 1, 1), ncol = 2),
  distortion_ratio = c(0.1, 0.1, 0.1, 0.1)
)

# Get file information and drop "id" column
n <- numeric(3)
x <- list()
for (i in 1:3) {
  n[i] <- nrow(sample_data[[i]])
  x[[i]] <- sample_data[[i]][, colnames(sample_data[[i]]) != "id", drop = FALSE]
}
N <- sum(n)

# Set Hyperparameters
hyper_beta <- matrix(
  rep(c(N * 0.1 * 0.01, N * 0.1), 4),
  ncol = 2, byrow = TRUE
)

hyper_sigma <- matrix(
  rep(c(0.01, 0.01), 2),
  ncol = 2, byrow = TRUE
)

# Perform record linkage using EVIL
result <- chomperEVIL(
  x = x,
  k = 3,  # number of datasets
  n = n,  # rows per dataset
  N = N,  # columns per dataset
  p = 4,  # fields per dataset
  M = c(5, 5),  # categories for discrete fields
  discrete_fields = c(1, 2),
  continuous_fields = c(3, 4),
  hyper_beta = hyper_beta,   # hyperparameter for distortion rate
  hyper_sigma = hyper_sigma, # hyperparameter for continuous fields
  n_threads = 4
)

# Performance evaluation
psm_ <- psm_vi(result$nu) # Calculate a posterior similarity matrix

# install.pakcages("salso")
library(salso)

salso_estimate <- salso(psm_,
  loss = binder(),
  maxZealousAttempts = 0, probSequentialAllocation = 1
) # Find a Bayes estimate that minimizes Binder's loss

linkage_structure <- list()
for (ll in seq_along(salso_estimate)) {
  linkage_structure[[ll]] <- which(salso_estimate == salso_estimate[ll])
}
linkage_estimation <- matrix(linkage_structure)

# install.packages("blink")
library(blink)

key_temp <- c()
for (i in 1:3) {
  key_temp <- c(key_temp, sample_data[[i]][, "id"])
}

truth_binded <- matrix(key_temp, nrow = 1)
linkage_structure_true <- links(truth_binded, TRUE, TRUE)
linkage_truth <- matrix(linkage_structure_true)

perf <- performance(linkage_estimation, linkage_truth, N)
print(perf)

Main Functions

Core Inference Functions

• chomperMCMC(): Markov Chain Monte Carlo
• chomperEVIL(): Evolutionary Variational Inference for record Linkage
• chomperCAVI(): Coordinate Ascent Variational Inference

Data Generation and Utilities

• generate_sample_data(): Create synthetic data for testing and validation
• flatten_posterior_samples(): Flatten posterior samples for obtaining a posterior similarity matrix

Evaluation and Performance

• psm_mcmc(): Posterior similarity matrix for MCMC results
• psm_vi(): Posterior similarity matrix for variational inference
• performance(): Evaluate performance of estimation

License

This package is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

Authors

• Hyungjoon Kim - Maintainer - GitHub
• Andee Kaplan - Contributor - GitHub
• Matthew Koslovsky - Contributor - GitHub.