MyNixOS website logo
Description

Creating Groups from Data.

Methods for dividing data into groups. Create balanced partitions and cross-validation folds. Perform time series windowing and general grouping and splitting of data. Balance existing groups with up- and downsampling or collapse them to fewer groups.

groupdata2

Author:Ludvig R. Olsen ( [email protected] )
License:MIT
Started: October 2016

CRAN_Status_Badge metacrandownloads minimal Rversion Codecov testcoverage GitHub Actions CIstatus AppVeyor buildstatus DOI

Overview

R package for dividing data into groups.

  • Create balanced partitions and cross-validation folds.
  • Perform time series windowing and general grouping and splitting of data.
  • Balance existing groups with up- and downsampling.
  • Collapse existing groups to fewer, balanced groups.
  • Finds values, or indices of values, that differ from the previous value by some threshold(s).
  • Check if two grouping factors have the same groups, memberwise.

Main functions

FunctionDescription
group_factor()Divides data into groups by a wide range of methods.
group()Creates grouping factor and adds to the given data frame.
splt()Creates grouping factor and splits the data by these groups.
partition()Splits data into partitions. Balances a given categorical variable and/or numerical variable between partitions and keeps all data points with a shared ID in the same partition.
fold()Creates folds for (repeated) cross-validation. Balances a given categorical variable and/or numerical variable between folds and keeps all data points with a shared ID in the same fold.
collapse_groups()Collapses existing groups into a smaller set of groups with categorical, numerical, ID, and size balancing.
balance()Uses up- and/or downsampling to equalize group sizes. Can balance on ID level. See wrappers: downsample(), upsample().

Other tools

FunctionDescription
all_groups_identical()Checks whether two grouping factors contain the same groups, memberwise.
differs_from_previous()Finds values, or indices of values, that differ from the previous value by some threshold(s).
find_starts()Finds values or indices of values that are not the same as the previous value.
find_missing_starts()Finds missing starts for the l_starts method.
summarize_group_cols()Calculates summary statistics about group columns (i.e. factors).
summarize_balances()Summarizes the balances of numeric, categorical, and ID columns in and between groups in one or more group columns.
ranked_balances()Extracts the standard deviations from the Summary data frame from the output of summarize_balances()
%primes%Finds remainder for the primes method.
%staircase%Finds remainder for the staircase method.

Table of Contents

Installation

CRAN version:

install.packages("groupdata2")

Development version:

install.packages("devtools")
devtools::install_github("LudvigOlsen/groupdata2")

Vignettes

groupdata2 contains a number of vignettes with relevant use cases and descriptions:

vignette(package = "groupdata2") # for an overview
vignette("introduction_to_groupdata2") # begin here

Data for examples

# Attach packages
library(groupdata2)
library(dplyr)       # %>% filter() arrange() summarize()
library(knitr)       # kable()
# Create small data frame
df_small <- data.frame(
  "x" = c(1:12),
  "species" = rep(c('cat', 'pig', 'human'), 4),
  "age" = sample(c(1:100), 12),
  stringsAsFactors = FALSE
)
# Create medium data frame
df_medium <- data.frame(
  "participant" = factor(rep(c('1', '2', '3', '4', '5', '6'), 3)),
  "age" = rep(c(20, 33, 27, 21, 32, 25), 3),
  "diagnosis" = factor(rep(c('a', 'b', 'a', 'b', 'b', 'a'), 3)),
  "diagnosis2" = factor(sample(c('x','z','y'), 18, replace = TRUE)),
  "score" = c(10, 24, 15, 35, 24, 14, 24, 40, 30, 
              50, 54, 25, 45, 67, 40, 78, 62, 30))
df_medium <- df_medium %>% arrange(participant)
df_medium$session <- rep(c('1','2', '3'), 6)

Functions

group_factor()

Returns a factor with group numbers, e.g. factor(c(1,1,1,2,2,2,3,3,3)).

This can be used to subset, aggregate, group_by, etc.

Create equally sized groups by setting force_equal = TRUE

Randomize grouping factor by setting randomize = TRUE

# Create grouping factor
group_factor(
  data = df_small, 
  n = 5, 
  method = "n_dist"
)
#>  [1] 1 1 2 2 3 3 3 4 4 5 5 5
#> Levels: 1 2 3 4 5

group()

Creates a grouping factor and adds it to the given data frame. The data frame is grouped by the grouping factor for easy use in magrittr (%>%) pipelines.

# Use group()
group(data = df_small, n = 5, method = 'n_dist') %>%
  kable()
xspeciesage.groups
1cat681
2pig391
3human12
4cat342
5pig873
6human433
7cat143
8pig824
9human594
10cat515
11pig855
12human215
# Use group() in a pipeline 
# Get average age per group
df_small %>%
  group(n = 5, method = 'n_dist') %>% 
  dplyr::summarise(mean_age = mean(age)) %>%
  kable()
.groupsmean_age
153.5
217.5
348.0
470.5
552.3
# Using group() with 'l_starts' method
# Starts group at the first 'cat', 
# then skips to the second appearance of "pig" after "cat",
# then starts at the following "cat".
df_small %>%
  group(n = list("cat", c("pig", 2), "cat"),
        method = 'l_starts',
        starts_col = "species") %>%
  kable()
xspeciesage.groups
1cat681
2pig391
3human11
4cat341
5pig872
6human432
7cat143
8pig823
9human593
10cat513
11pig853
12human213

splt()

Creates the specified groups with group_factor() and splits the given data by the grouping factor with base::split. Returns the splits in a list.

splt(data = df_small,
     n = 3,
     method = 'n_dist') %>%
  kable()
xspeciesage
1cat68
2pig39
3human1
4cat34
xspeciesage
55pig87
66human43
77cat14
88pig82
xspeciesage
99human59
1010cat51
1111pig85
1212human21

partition()

Creates (optionally) balanced partitions (e.g. training/test sets). Balance partitions on categorical variable(s) and/or a numerical variable. Make sure that all datapoints sharing an ID is in the same partition.

# First set seed to ensure reproducibility
set.seed(1)

# Use partition() with categorical and numerical balancing,
# while ensuring all rows per ID are in the same partition
df_partitioned <- partition(
  data = df_medium, 
  p = 0.7,
  cat_col = 'diagnosis',
  num_col = "age",
  id_col = 'participant'
)

df_partitioned %>% 
  kable()
participantagediagnosisdiagnosis2scoresession
120az101
120ay242
120ax453
233bz241
233bx402
233bx673
327az151
327ax302
327az403
421bz351
421bx502
421bz783
participantagediagnosisdiagnosis2scoresession
532by241
532bx542
532bz623
625ax141
625az252
625ax303

fold()

Creates (optionally) balanced folds for use in cross-validation. Balance folds on categorical variable(s) and/or a numerical variable. Ensure that all datapoints sharing an ID is in the same fold. Create multiple unique fold columns at once, e.g. for repeated cross-validation.

# First set seed to ensure reproducibility
set.seed(1)

# Use fold() with categorical and numerical balancing,
# while ensuring all rows per ID are in the same fold
df_folded <- fold(
  data = df_medium, 
  k = 3,
  cat_col = 'diagnosis',
  num_col = "age",
  id_col = 'participant'
)

# Show df_folded ordered by folds
df_folded %>% 
  arrange(.folds) %>%
  kable()
participantagediagnosisdiagnosis2scoresession.folds
120az1011
120ay2421
120ax4531
532by2411
532bx5421
532bz6231
421bz3512
421bx5022
421bz7832
625ax1412
625az2522
625ax3032
233bz2413
233bx4023
233bx6733
327az1513
327ax3023
327az4033
# Show distribution of diagnoses and participants
df_folded %>% 
  group_by(.folds) %>% 
  count(diagnosis, participant) %>% 
  kable()
.foldsdiagnosisparticipantn
1a13
1b53
2a63
2b43
3a33
3b23
# Show age representation in folds
# Notice that we would get a more even distribution if we had more data.
# As age is fixed per ID, we only have 3 ages per category to balance with.
df_folded %>% 
  group_by(.folds) %>% 
  summarize(mean_age = mean(age),
            sd_age = sd(age)) %>% 
  kable()
.foldsmean_agesd_age
1266.57
2232.19
3303.29

Notice, that the we now have the opportunity to include the session variable and/or use participant as a random effect in our model when doing cross-validation, as any participant will only appear in one fold.

We also have a balance in the representation of each diagnosis, which could give us better, more consistent results.

collapse_groups()

Collapses a set of groups into a smaller set of groups while attempting to balance the new groups by specified numerical columns, categorical columns, level counts in ID columns, and/or the number of rows.

# We consider each participant a group
# and collapse them into 3 new groups
# We balance the number of levels in diagnosis2 column, 
# as this diagnosis is not constant within the participants
df_collapsed <- collapse_groups(
  data = df_medium,
  n = 3,
  group_cols = 'participant',
  cat_cols = 'diagnosis2',
  num_cols = "score"
) 

# Show df_collapsed ordered by new collapsed groups
df_collapsed %>% 
  arrange(.coll_groups) %>%
  kable()
participantagediagnosisdiagnosis2scoresession.coll_groups
120az1011
120ay2421
120ax4531
233bz2411
233bx4021
233bx6731
327az1512
327ax3022
327az4032
421bz3512
421bx5022
421bz7832
532by2413
532bx5423
532bz6233
625ax1413
625az2523
625ax3033

# Summarize the balances of the new groups
coll_summ <- df_collapsed %>% 
  summarize_balances(group_cols = '.coll_groups',
                     cat_cols = "diagnosis2",
                     num_cols = "score")

coll_summ$Groups %>% 
  kable()
.group_col.group# rowsmean(score)sum(score)# diag_x# diag_y# diag_z
.coll_groups1635.0210312
.coll_groups2641.3248204
.coll_groups3634.8209312

coll_summ$Summary %>% 
  kable()
.group_colmeasure# rowsmean(score)sum(score)# diag_x# diag_y# diag_z
.coll_groupsmean637.06222.32.6670.6672.67
.coll_groupsmedian635.00210.03.0001.0002.00
.coll_groupsSD03.7122.20.5770.5771.16
.coll_groupsIQR03.2519.50.5000.5001.00
.coll_groupsmin634.83209.02.0000.0002.00
.coll_groupsmax641.33248.03.0001.0004.00

# Check the across-groups standard deviations 
# This is a measure of how balanced the groups are (lower == more balanced)
# and is especially useful when comparing multiple group columns
coll_summ %>% 
  ranked_balances() %>%
  kable()
.group_colmeasure# rowsmean(score)sum(score)# diag_x# diag_y# diag_z
.coll_groupsSD03.7122.20.5770.5771.16

Recommended: By enabling the auto_tune setting, we often get a much better balance.

balance()

Uses up- and/or downsampling to fix the group sizes to the min, max, mean, or median group size or to a specific number of rows. Balancing can also happen on the ID level, e.g. to ensure the same number of IDs in each category.

# Lets first unbalance the dataset by removing some rows
df_b <- df_medium %>% 
  arrange(diagnosis) %>% 
  filter(!row_number() %in% c(5,7,8,13,14,16,17,18))

# Show distribution of diagnoses and participants
df_b %>% 
  count(diagnosis, participant) %>% 
  kable()
diagnosisparticipantn
a13
a32
a61
b23
b41
# First set seed to ensure reproducibility
set.seed(1)

# Downsampling by diagnosis
balance(
  data = df_b, 
  size = "min", 
  cat_col = "diagnosis"
) %>% 
  count(diagnosis, participant) %>% 
  kable()
diagnosisparticipantn
a12
a31
a61
b23
b41
# Downsampling the IDs
balance(
  data = df_b, 
  size = "min", 
  cat_col = "diagnosis", 
  id_col = "participant", 
  id_method = "n_ids"
) %>% 
  count(diagnosis, participant) %>% 
  kable()
diagnosisparticipantn
a13
a32
b23
b41

Grouping Methods

There are currently 10 methods available. They can be divided into 6 categories.

Examples of group sizes are based on a vector with 57 elements.

Specify group size

Method: greedy

Divides up the data greedily given a specified group size.

E.g. group sizes: 10, 10, 10, 10, 10, 7

Specify number of groups

Method: n_dist (Default)

Divides the data into a specified number of groups and distributes excess data points across groups.

E.g. group sizes: 11, 11, 12, 11, 12

Method: n_fill

Divides the data into a specified number of groups and fills up groups with excess data points from the beginning.

E.g. group sizes: 12, 12, 11, 11, 11

Method: n_last

Divides the data into a specified number of groups. The algorithm finds the most equal group sizes possible, using all data points. Only the last group is able to differ in size.

E.g. group sizes: 11, 11, 11, 11, 13

Method: n_rand

Divides the data into a specified number of groups. Excess data points are placed randomly in groups (only 1 per group).

E.g. group sizes: 12, 11, 11, 11, 12

Specify list

Method: l_sizes

Uses a list / vector of group sizes to divide up the data.
Excess data points are placed in an extra group.

E.g. n = c(11, 11) returns group sizes: 11, 11, 35

Method: l_starts

Uses a list of starting positions to divide up the data.
Starting positions are values in a vector (e.g. column in data frame). Skip to a specific nth appearance of a value by using c(value, skip_to).

E.g. n = c(11, 15, 27, 43) returns group sizes: 10, 4, 12, 16, 15

Identical to n = list(11, 15, c(27, 1), 43 where 1 specifies that we want the first appearance of 27 after the previous value 15.

If passing n = "auto" starting positions are automatically found with find_starts().

Specify distance between members

Method: every

Every nth data point is combined to a group.

E.g. group sizes: 12, 12, 11, 11, 11

Specify step size

Method: staircase

Uses step_size to divide up the data. Group size increases with 1 step for every group, until there is no more data.

E.g. group sizes: 5, 10, 15, 20, 7

Specify start at

Method: primes

Creates groups with sizes corresponding to prime numbers.
Starts at n (prime number). Increases to the the next prime number until there is no more data.

E.g. group sizes: 5, 7, 11, 13, 17, 4

Balancing ID Methods

There are currently 4 methods for balancing (up-/downsampling) on ID level in balance().

ID method: n_ids

Balances on ID level only. It makes sure there are the same number of IDs in each category. This might lead to a different number of rows between categories.

ID method: n_rows_c

Attempts to level the number of rows per category, while only removing/adding entire IDs. This is done with repetition and by iteratively picking the ID with the number of rows closest to the lacking/excessive number of rows in the category.

ID method: distributed

Distributes the lacking/excess rows equally between the IDs. If the number to distribute cannot be equally divided, some IDs will have 1 row more/less than the others.

ID method: nested

Balances the IDs within their categories, meaning that all IDs in a category will have the same number of rows.

Metadata

Version

2.0.3

License

Unknown

Platforms (75)

    Darwin
    FreeBSD
    Genode
    GHCJS
    Linux
    MMIXware
    NetBSD
    none
    OpenBSD
    Redox
    Solaris
    WASI
    Windows
Show all
  • aarch64-darwin
  • aarch64-genode
  • aarch64-linux
  • aarch64-netbsd
  • aarch64-none
  • aarch64_be-none
  • arm-none
  • armv5tel-linux
  • armv6l-linux
  • armv6l-netbsd
  • armv6l-none
  • armv7a-darwin
  • armv7a-linux
  • armv7a-netbsd
  • armv7l-linux
  • armv7l-netbsd
  • avr-none
  • i686-cygwin
  • i686-darwin
  • i686-freebsd
  • i686-genode
  • i686-linux
  • i686-netbsd
  • i686-none
  • i686-openbsd
  • i686-windows
  • javascript-ghcjs
  • loongarch64-linux
  • m68k-linux
  • m68k-netbsd
  • m68k-none
  • microblaze-linux
  • microblaze-none
  • microblazeel-linux
  • microblazeel-none
  • mips-linux
  • mips-none
  • mips64-linux
  • mips64-none
  • mips64el-linux
  • mipsel-linux
  • mipsel-netbsd
  • mmix-mmixware
  • msp430-none
  • or1k-none
  • powerpc-netbsd
  • powerpc-none
  • powerpc64-linux
  • powerpc64le-linux
  • powerpcle-none
  • riscv32-linux
  • riscv32-netbsd
  • riscv32-none
  • riscv64-linux
  • riscv64-netbsd
  • riscv64-none
  • rx-none
  • s390-linux
  • s390-none
  • s390x-linux
  • s390x-none
  • vc4-none
  • wasm32-wasi
  • wasm64-wasi
  • x86_64-cygwin
  • x86_64-darwin
  • x86_64-freebsd
  • x86_64-genode
  • x86_64-linux
  • x86_64-netbsd
  • x86_64-none
  • x86_64-openbsd
  • x86_64-redox
  • x86_64-solaris
  • x86_64-windows