shamochu

“Shuffle and merge overlapping chunks” lossless compression.

Shamochu is short for “Shuffle and merge overlapping chunks” lossless compression.

The idea for the compression is:

1. Split the input array in chunks of a given size.

2. Rearrange the order of the chunks in order to optimize consecutive chunks overlaps.

3. Create a data table with the reordered chunks and an index table that maps the original chunk index to its offset in the data table.

Then the data can be accessed in 𝒪(1) via a few bitwise operations and indexing two arrays.

The same operation can then be applied to the index table and may lead to further compression.

Trivial example (chunk size: 4):

  [1, 2, 3, 4, 2, 3, 4, 5, 0, 1, 2, 3]          # source data
  -> [[1, 2, 3, 4], [2, 3, 4, 5], [0, 1, 2, 3]] # make chunks
  -> [[0, 1, 2, 3], [1, 2, 3, 4], [2, 3, 4, 5]] # rearrange to have best overlaps
  -> {data: [0, 1, 2, 3, 4, 5], offsets: [1, 2, 0]} # overlap chunks & compute
                                                    # their offsets

Then we can retrieve the data from the original array at index i with the following formula:

    mask = (1 << chunk_size) - 1
    original[i] = data[offsets[i >> chunk_size] + (i & mask)]

Since the index array is itself quite repetitive with the real data, we can apply the compression a second time to the offsets table.

The complete algorithm optimizes the chunk sizes for both arrays in order to get the lowest total data size. Given the chunks sizes cs_data and cs_offsets:

1. We compute the corresponding masks:

   • mask_data = (1 << cs_data) - 1 and

   • mask_offsets = (1 << cs_offsets) - 1.

2. We can retrieve the original value at index k with the following formula:

    data[
        offsets1[
            offsets2[i >> (cs_data + cs_offsets)] +
            ((i >> cs_data) & mask_offsets)
        ] +
        (i & mask_data)
    ];

Notes

This work took inspiration from “Fast character case conversion… or how to really compress sparse arrays” by Alexander Pankratov.