Description
Safetensors File Format.
Description
A file format for storing tensors that is secure (doesn't allow for code execution), fast and simple to implement. 'safetensors' also enables cross language and cross frameworks compatibility making it an ideal format for storing machine learning model weights.
README.md
safetensors
safetensors is a pure R implementation of the safetensors file format.
Currently only reading files is supported.
Installation
safetensors can be installed from CRAN with:
install.packages("safetensors")
The development version of safetensors from GitHub with:
# install.packages("devtools")
devtools::install_github("mlverse/safetensors")
Example
Here’s an example of writing and reading safetensors files:
library(torch)
library(safetensors)
tensors <- list(
x = torch_randn(10, 10),
y = torch_ones(10, 10)
)
str(tensors)
#> List of 2
#> $ x:Float [1:10, 1:10]
#> $ y:Float [1:10, 1:10]
tmp <- tempfile()
safe_save_file(tensors, tmp)
tensors <- safe_load_file(tmp)
str(tensors)
#> List of 2
#> $ x:Float [1:10, 1:10]
#> $ y:Float [1:10, 1:10]
#> - attr(*, "metadata")=List of 2
#> ..$ x:List of 3
#> .. ..$ shape : int [1:2] 10 10
#> .. ..$ dtype : chr "F32"
#> .. ..$ data_offsets: int [1:2] 0 400
#> ..$ y:List of 3
#> .. ..$ shape : int [1:2] 10 10
#> .. ..$ dtype : chr "F32"
#> .. ..$ data_offsets: int [1:2] 400 800
#> - attr(*, "max_offset")= int 929