Retrieve, Transform and Analyze the Barcode of Life Data Systems Data.
BOLDconnectR is a package designed for retrieval, transformation and analysis of the data available in the Barcode Of Life Data Systems (BOLD) database. This package provides the functionality to obtain public and private user data available in the database in the Barcode Core Data Model (BCDM) format. Data include information on the taxonomy,geography,collection,identification and DNA barcode sequence of every submission. The manual is currently hosted here (https://github.com/boldsystems-central/BOLDconnectR_examples/blob/main/BOLDconnectR_1.0.0.pdf)
BOLDconnectR requires R version 4.0 or above to function properly. The versions of dependent packages have also been set such that they would work with R >= 4.0. In addition, there are a few suggested packages that are not mandatory for the package to download and install properly, but, are essential for a couple of functions to work. The names and exact versions of the dependencies/suggestions are given here (https://github.com/boldsystems-central/BOLDconnectR/blob/main/DESCRIPTION). More details on Suggested packages provided below.
Installation
The package can be installed using devtools::install_github function from the devtools package in R (which needs to be installed before installing BOLDConnectR).
devtools::install_github("https://github.com/boldsystems-central/BOLDconnectR")
library(BOLDconnectR)
BOLDconnectR has 11 functions currently:
- bold.fields.info
- bold.apikey
- bold.fetch
- bold.public.search
- bold.full.search
- bold.data.summarize
- bold.analyze.align
- bold.analyze.tree
- bold.analyze.diversity
- bold.analyze.map
- bold.export
Note on Suggested packagesFunction 6: bold.data.summarize requires the packages Biostrings to be installed and imported in R session beforehand for generating the barcode_summary. Function 7: bold.analyze.align requires the packages msa and Biostrings to be installed and imported in the R session beforehand. Function 8 also uses the output generated from function 7. msa and Biostrings can be installed using the BiocManager package.
if (!requireNamespace("BiocManager", quietly=TRUE))
install.packages("BiocManager")
BiocManager::install("msa")
BiocManager::install("Biostrings")
library(msa)
library(Biostrings)
Note on API key
The function bold.fetch requires an api key internally to access and download all public + private user data. The API key needed to retrieve BOLD records is found in the BOLD ‘Workbench’ https://bench.boldsystems.org/index.php/Login/page?destination=MAS_Management_UserConsole. After logging in, navigate to ‘Your Name’ (located at the top left-hand side of the window) and click ‘Edit User Preferences’. You can find the API key in the ‘User Data’ section. Please note that to have an API key available in the workbench, a user must have uploaded at least 10,000 records to BOLD. API key can be saved in the R session using bold.apikey() function. Please note that the API keys are regenerated periodically and will be updated in the user's workbench account. Using old keys will result in a HTTP 401 error.
# Substitute ‘00000000-0000-0000-0000-000000000000’ with your key
# bold.apikey(‘00000000-0000-0000-0000-000000000000’)
Basic usage of BOLDConnectR functionality
API key function must be run prior to using the fetch function (Please see above).
Fetch data
BCDM_data<-bold.fetch(get_by = "processid",
identifiers = test.data$processid)
#> [32mInitiating download[0m
#> [31m Downloading data in a single batch [0m
#> [32mDownload complete & BCDM dataframe generated[0m
knitr::kable(head(BCDM_data,4))
| processid | record_id | insdc_acs | sampleid | specimenid | taxid | short_note | identification_method | museumid | fieldid | collection_code | processid_minted_date | inst | funding_src | sex | life_stage | reproduction | habitat | collectors | site_code | specimen_linkout | collection_event_id | sampling_protocol | tissue_type | collection_date_start | collection_time | associated_taxa | associated_specimens | voucher_type | notes | taxonomy_notes | collection_notes | geoid | marker_code | kingdom | phylum | class | order | family | subfamily | tribe | genus | species | subspecies | identification | identification_rank | species_reference | identified_by | sequence_run_site | nuc | nuc_basecount | sequence_upload_date | bin_uri | bin_created_date | elev | depth | coord | coord_source | coord_accuracy | elev_accuracy | depth_accuracy | realm | biome | ecoregion | region | sector | site | country_iso | country.ocean | province.state | bold_recordset_code_arr | collection_date_end |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SSWLD6460-13 | SSWLD6460-13.COI-5P | KM825932 | BIOUG06662-C01 | 3435715 | 9199 | Waterton Lakes NP | BOLD ID Engine: top hits | BIOUG06662-C01 | L#12BIOBUS-1587 | BIOUG | 2013-07-04 | Centre for Biodiversity Genomics | iBOL:WG1.9 | NA | NA | NA | Forest | BIOBus 2012 | BIOUG:WATERTON-NP:2 | NA | NA | Sweep Net | Whole Voucher | 2012-08-08 | NA | NA | NA | Vouchered:Registered Collection | NA | NA | 5 min sweep x4 collectors (2)|Sunny with slight haze, 23C|montane forest, douglas fir and lodgepole pine stand with aspen and birch understory | 533 | COI-5P | Animalia | Arthropoda | Arachnida | Araneae | Salticidae | NA | NA | Eris | Eris militaris | NA | Eris militaris | species | (Hentz, 1845) | Monica R. Young | Centre for Biodiversity Genomics | AACGTTATATTTAATTTTTGGAGCTTGATCAGCTATAGTTGGTACTGCTATAAGAGTATTAATTCGAATAGAATTAGGACAAACT—GGATCATTTTTAGGT————AATGATCATATATATAATGTAATTGTAACTGCTCATGCTTTTGTAATGATTTTTTTTATAGTAATACCAATTATAATTGGGGGATTTGGTAATTGGTTAGTTCCTTTAATGTTAGGGGCTCCGGATATAGCTTTTCCTCGAATAAATAATTTAAGTTTTTGATTATTACCTCCTTCTTTATTTTTATTGTTTATTTCTTCTATAGCTGAAATAGGGGTT—GGAGCTGGATGAACAGTATATCCTCCTTTGGCATCTATTGTTGGACATAATGGTAGATCAGTAGATTTTGCTATTTTTTCTTTACATTTAGCTGGTGCTTCATCAATTATAGGAGCTATTAATTTTATTTCTACTATTATTAATATACGA—TCAGTAGGAATATCTTTAGATAAAATTCCTTTATTTGTTTGATCTGTAATAATTACTGCTGTATTATTATTGTTATCATTACCTGTTTTAGCAGGAGCTATTACTATATTATTAACTGAT——————— | 589 | 2013-09-16 | BOLD:AAA5654 | 2010-07-15 | 1562 | NA | 49.065,-113.778 | GPSmap 60Cx | NA | NA | NA | Nearctic | NA | Northern_Rockies_conifer_forests | Waterton Lakes National Park | East of 2 Flags Lookout | Highway 6 pulloff | CA | Canada | Alberta | SSWLD,DS-MOB113,DS-BICNP02,DS-SOC2014,DS-ARANCCYH,DATASET-BBWLNP1,DS-SPCANADA,DS-JUMPGLOB,DS-MOB112,DS-CANSS | NA |
| SPITO327-14 | SPITO327-14.COI-5P | KP654265 | BIOUG12602-G11 | 4610196 | 9162 | NA | NA | BIOUG12602-G11 | L#14BLITZ-001 | BIOUG | 2014-06-10 | Centre for Biodiversity Genomics | iBOL:WG1.9 | M | A | S | NA | Gergin Blagoev | NA | NA | NA | Free Hand Collection | NA | 2014-05-24 | NA | NA | NA | museum voucher | Collected May 24-14, as part of Humber Watershed BioBlitz | NA | NA | 528 | COI-5P | Animalia | Arthropoda | Arachnida | Araneae | Salticidae | NA | NA | Phidippus | Phidippus audax | NA | Phidippus audax | species | (Hentz, 1845) | Gergin A. Blagoev | Centre for Biodiversity Genomics | -ACATTATATTTGATTTTTGGAGCTTGGGCTGCAATAGTTGGTACTGCAATA—AGTGTATTGATTCGAATAGAATTGGGTCAAACTGGATCATTTATAGGAAAT—GATCATATATATAATGTAATTGTGACTGCTCATGCTTTTGTTATAATTTTTTTTATAGTAATACCTATTATGATTGGAGGATTTGGAAACTGATTAGTTCCTTTAATA—TTAGGTGCTCCTGATATGGCTTTTCCTCGTATAAATAATTTGAGATTTTGATTATTACCCCCTTCTTTATTTTTATTATTTATTTCTTCCATAGCTGAGGTAGGTGTAGGGGCTGGTTGGACAGTTTATCCACCTTTGGCCTCTATTGTTGGGCATAATGGAAGATCAGTAGATTTT—GCTATTTTTTCATTACATTTAGCTGGTGCTTCATCAATTATAGGAGCTATTAATTTTATTTCTACAATTATTAATATACGTTCTTTAGGAATGTCTTTAGATAAAATTCCTTTGTTTGTTTGATCTGTAATAATTACTGCAGTTTTGTTATTACTTTCTCTTCCTGTATTAGCTGGG—GCTATTACTATATTGTTGACTGAT——————————————————————————————————————————————————————————————————————————————————————- | 588 | 2014-06-27 | BOLD:AAC6891 | 2010-07-15 | 380 | NA | 43.933,-79.928 | NA | NA | NA | NA | Nearctic | NA | Eastern_Great_Lakes_lowland_forests | Humber Watershed | NA | Glen Haffy Conservation Area | CA | Canada | Ontario | SPITO,DS-SOC2014,DS-ARANCCYH,DS-TMPSRCH,DS-SPCANADA,DS-OLOCC2,DS-JUMPGLOB | NA |
| ARONT071-09 | ARONT071-09.COI-5P | GU682836 | 09ONTGAB-183 | 1229966 | 30494 | SPIOH09-1 F11 | NA | 09ONTGAB-183 | 090816FH | BIOUG | 2009-09-23 | Centre for Biodiversity Genomics | iBOL:WG1.9 | M | I | S | NA | G.A.Blagoev | NA | NA | NA | NA | 2009-08-16 | NA | NA | NA | NA | NA | NA | 528 | COI-5P | Animalia | Arthropoda | Arachnida | Araneae | Salticidae | NA | NA | Naphrys | Naphrys pulex | NA | Naphrys pulex | species | (Hentz, 1846) | Gergin A. Blagoev | Centre for Biodiversity Genomics | AACATTATATTTGATTTTTGGTGCTTGATCAGCTATAGTAGGTACGGCTATAAGAGTTTTGATTCGAATAGAGTTGGGACAGACTGGTAATTTTTTGGGAAATGATCATTTATATAATGTCATTGTAACTGCTCATGCTTTTGTTATGATTTTTTTTATAGTAATACCTATTTTGATTGGTGGTTTTGGTAATTGATTAGTGCCATTAATATTAGGGGCTCCTGATATAGCTTTTCCTCGGATGAATAATTTGAGATTTTGGTTATTACCCCCTTCATTAATACTCTTATTTATATCTTCAATAGTGGAGATAGGGGTAGGAGCAGGGTGAACAGTGTATCCCCCATTAGCTTCTGTTGTAGGTCATAATGGAAGATCTGTTGATTTTGCTATTTTTTCTTTACATTTAGCGGGGGCTTCTTCTATTATAGGAGCAGTTAATTTTATTTCTACTATTATTAATATACGTGTATTAGGAATGAGAATAGATAAGATTCCTTTGTTTGTTTGGTCAGTTGGGATTACTGCTGTATTATTATTATTATCACTACCAGTGTTGGCTGGTGCTATTACAATATTGTTGACTGATCGTAATTTTAATACCTCTTTTTTTGATCCTGCGGGAGGAGGGGATCCGGTTTTGTTTCAGCATTTATTT | 658 | 2009-10-29 | BOLD:AAC2433 | 2010-07-15 | 300 | NA | 43.691,-80.414 | NA | NA | NA | Nearctic | NA | Eastern_Great_Lakes_lowland_forests | Wellington Co. | Elora | Beach | CA | Canada | Ontario | ARONT,DS-MOB113,DS-SOC2014,DS-MYBCA,DS-ARANCCYH,DS-SPCANADA,DS-OLOCC1,DS-JUMPGLOB,DS-MOB112,DS-JALPHA | NA | |||
| SPIRU1237-11 | SPIRU1237-11.COI-5P | KF368796 | BIOUG00629-G03 | 1982513 | 842900 | ocean beach|AP|HC | NA | BIOUG00629-G03 | L#10PROBE-6510 | 10PROBE | 2011-05-16 | Centre for Biodiversity Genomics | iBOL:WG1.10 | I | S | NA | V. Junea | BIOUG:Churchill | NA | NA | NA | NA | 2010-07-30 | NA | NA | NA | whole specimen | NA | NA | NA | 531 | COI-5P | Animalia | Arthropoda | Arachnida | Araneae | Salticidae | NA | NA | Sittisax | Sittisax ranieri | NA | Sittisax ranieri | species | (G. W. Peckham & E. G. Peckham, 1909) | Gergin A. Blagoev | Centre for Biodiversity Genomics | TACGTTATATTTAGTTTTTGGAGCTTGGTCTGCTATAGTTGGTACGGCTATAAGAGTTTTAATTCGTATAGAATTAGGTCAAACTGGTCATTTTTTAGGAAATGATCATTTGTATAATGTAATTGTTACTGCACATGCATTTGTTATAATTTTTTTTATAGTAATACCTATTTTGATTGGAGGTTTTGGTAATTGATTAGTCCCTCTAATGTTAGGAGCTCCGGATATAGCTTTTCCTCGTATAAATAATTTAAGTTTTTGATTATTACCTCCTTCATTATTTTTATTATTTATTTCATCTATAGCTGAGATAGGAGTAGGGGCAGGGTGAACTGTTTATCCTCCATTAGCTTCTATTGTAGGTCATAATGGAAGTTCGGTAGATTTTGCTATTTTTTCTCTTCATTTGGCTGGGGCTTCATCAATTATAGGTGCTATTAATTTTATTTCAACTGTTATTAATATACGATCGGTGGGTATATCAATAGATAAGATTCCATTGTTTGTTTGGTCTGTTGTAATTACTGCTGTATTATTGTTATTGTCTTTACCTGTTTTAGCGGGTGCAATTACTATGCTATTGACTGATCGAAATTTTAATACGTCTTTTTTTGATCCTGCTGGAGGAGGGGATCCAATTTTATTTCAACATTTATTT | 658 | 2012-11-23 | BOLD:AAC2061 | 2010-07-15 | NA | NA | 58.772,-93.843 | GPS WGS84 | NA | NA | NA | Nearctic | NA | Southern_Hudson_Bay_taiga | Churchill | 16 km E Churchill, Bird Cove, Rock Bluff A | Beach | CA | Canada | Manitoba | CHSPI,DATASET-CHURCH12,DS-MOB113,DS-SOC2014,DS-ARANCCYH,DS-TMPSRCH,DS-SPCANADA,DS-ATBIB,DS-JUMPGLOB,DS-MOB112,DS-ARA43210 | NA |
Similarly, sampleids or dataset_codes or project_codes can also be used to fetch data. The data can also be filtered on different parameters such as Geography, Attributions and DNA Sequence information using the _filt arguments available in the function
Summarize downloaded data
Downloaded data can then be summarized in different ways. Options currently include a concise summary of all the data, detailed taxonomic counts, data completeness and a barcode-based summary
BCDM_data_summary<-bold.data.summarize(bold_df = BCDM_data,
summary_type = "concise_summary")
BCDM_data_summary$concise_summary
#> Category Value
#> 1 Total_records 1336
#> 2 Total_records_w_sequences 1336
#> 3 Unique_species 80
#> 4 Unique_BINs 117
#> 5 Unique_countries 1
#> 6 Unique_institutes 6
#> 7 Unique_identified_by 7
#> 8 Unique_specimen_depositories 3
#> 9 Unique_markers COI-5P
#> 10 Amplicon_length_range 508-658
A concise summary providing a high level overview of the data
Export the downloaded data
Downloaded data can also be exported to the local machine either as a flat file or as a FASTA file for any third party sequence analysis tools.The flat file contents can be modified as per user requirements (entire data or specific presets or individual fields).
# Preset dataframe
# bold.export(bold_df = BCDM_data,
# export_type = "preset_df",
# presets = 'taxonomy',
# cols_for_fas_names = NULL,
# export = "file_path_with_intended_name.csv")
# Unaligned fasta file
# bold.export(bold_df = BCDM_data,
# export_type = "fas",
# cols_for_fas_names = c("bin_uri","genus","species"),
# export = "file_path_with_intended_name.fas")
Other functions
The package also has functions that provide sequence alignment, NJ clustering, biodiversity analysis and occurrence mapping using the downloaded BCDM data. Additionally, some of these functions also output objects that are commonly used by other R packages (Ex. ‘sf’ dataframe, occurrence matrix for ‘vegan’ and ‘betapart’). Please go through the help manual (Link provided above) for detailed usage of all the functions of BOLDConnectR with examples.
BOLDconnectR can retrieve data very fast (~100k records in a minute on a fast wired connection).
Citation: Padhye SM, Ballesteros-Mejia CL, Agda TJA, Agda JRA, Ratnasingham S. BOLDconnectR: An R package for streamlined retrieval, transformation and analysis of BOLD DNA barcode data (MS in prep).