Large and heterogeneous datasets may contain thousands of records missing spatial or taxonomic information (partially or entirely) as well as several records outside a region of interest or from doubtful sources. Such lower quality data are not fit for use in many research applications without prior amendments. The ‘Pre-filter’ module contains a series of tests to detect, remove, and, whenever, possible, correct such erroneous or suspect records.
Check here how to install the bdc package.
Read the merged database created in the module Standardization and integration of different datasets of the bdc package. It is also possible to read any datasets containing the required fields to run the functions (more details here).
The results of the VALIDATION test used to flag data quality are appended in separate fields in this database and retrieved as TRUE ( ✅ ok ) or FALSE (❌check carefully ).
VALIDATION. Flag records missing species names.
check_pf <- bdc_scientificName_empty( data = database, sci_name = "scientificName") #> bdc_scientificName_empty: #> Flagged 5 records. #> One column was added to the database.
VALIDATION. Flag records missing partial or complete information on geographic coordinates.
check_pf <- bdc_coordinates_empty( data = check_pf, lat = "decimalLatitude", lon = "decimalLongitude") #> bdc_coordinates_empty: #> Flagged 44 records. #> One column was added to the database.
VALIDATION. This test flags records with out-of-range coordinates: latitude > 90 or -90; longitude >180 or -180.
check_pf <- bdc_coordinates_outOfRange( data = check_pf, lat = "decimalLatitude", lon = "decimalLongitude") #> bdc_coordinates_outOfRange: #> Flagged 1 records. #> One column was added to the database.
VALIDATION. This test flags records from doubtful source. For example, records from drawings, photographs, or multimedia objects, fossil records, among others.
# Check record sources of your dataset using: # check_pf %>% # dplyr::group_by(basisOfRecord) %>% # dplyr::summarise(n = dplyr::n()) check_pf <- bdc_basisOfRecords_notStandard( data = check_pf, basisOfRecord = "basisOfRecord", names_to_keep = "all") #> bdc_basisOfRecords_notStandard: #> Flagged 0 of the following specific nature: #> character(0) #> One column was added to the database.
ENRICHMENT. Deriving country names for records missing country names.
check_pf <- bdc_country_from_coordinates( data = check_pf, lat = "decimalLatitude", lon = "decimalLongitude", country = "country") #> bdc_country_from_coordinates: #> Country names were added to 27 records.
ENRICHMENT. Country names are standardized against a list of country names in several languages retrieved from International Organization for Standardization https://stefangabos.github.io/world_countries/.
check_pf <- bdc_country_standardized( data = check_pf, country = "country" ) #> Standardizing country names #> #> country found: Bolivia #> country found: Brazil #> |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s #> |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s #> |++++++++++++++++++++++++++++++++++++++++++++++++++| 100% elapsed=00s #> #> bdc_country_standardized: #> The country names of 65 records were standardized. #> Two columns ('country_suggested' and 'countryCode') were added to the database.
AMENDMENT. The mismatch between informed country and coordinates can be the result of negative or transposed coordinates. Once detected a mismatch, different coordinate transformations are made to correct the country and coordinates mismatch. Verbatim coordinates are then replaced by the rectified ones in the returned database (a database containing verbatim and corrected coordinates can be created in the “Output” folder if
save_outputs = TRUE). Records near countries coastline are not tested to avoid incur in false positives.
check_pf <- bdc_coordinates_transposed( data = check_pf, id = "database_id", sci_names = "scientificName", lat = "decimalLatitude", lon = "decimalLongitude", country = "country_suggested", countryCode = "countryCode", border_buffer = 0.2, # in decimal degrees (~22 km at the equator) save_outputs = FALSE ) #> Correcting latitude and longitude transposed #> #> 15 occurrences will be tested #> Processing occurrences from: BR (15) #> No latitude and longitude were transposed
VALIDATION. Records outside one or multiple reference countries; i.e., records in other countries or at an informed distance from the coast (e.g., in the ocean). This last step avoids flagging as invalid records close to country limits (e.g., records of coast or marshland species).
check_pf <- bdc_coordinates_country_inconsistent( data = check_pf, country_name = "Brazil", country = "country_suggested", lon = "decimalLongitude", lat = "decimalLatitude", dist = 0.1 # in decimal degrees (~11 km at the equator) ) #> dist is assumed to be in decimal degrees (arc_degrees). #> although coordinates are longitude/latitude, st_intersection assumes that they are planar #> #> bdc_coordinates_country_inconsistent: #> Flagged 15 records. #> One column was added to the database.
ENRICHMENT. Coordinates can be derived from a detailed description of the locality associated with records in a process called retrospective geo-referencing.
xyFromLocality <- bdc_coordinates_from_locality( data = check_pf, locality = "locality", lon = "decimalLongitude", lat = "decimalLatitude", save_outputs = FALSE ) #> bdc_coordinates_from_locality #> Found 38 records missing or with invalid coordinates but with potentially useful information on locality
Here we create a column named “.summary” summing up the results of all VALIDATION tests (those starting with “.”). This column is FALSE when a record is flagged as FALSE in any data quality test (i.e. potentially invalid or suspect record).
check_pf <- bdc_summary_col(data = check_pf) #> bdc_summary_col: #> Flagged 65 records. #> One column was added to the database.
Creating a report summarizing the results of all tests. The report can be automatically saved if
save_report = TRUE.
report <- bdc_create_report(data = check_pf, database_id = "database_id", workflow_step = "prefilter", save_report = FALSE) report
Here we create figures (bar plots and maps) to make the interpretation of the results of data quality tests easier. See some examples below. Figures can be automatically saved if
save_figures = TRUE.
figures <- bdc_create_figures(data = check_pf, database_id = "database_id", workflow_step = "prefilter", save_figures = FALSE) # Check figures using figures$.coordinates_country_inconsistent
We can remove records flagged as erroneous or suspect to obtain a “clean” database. Records missing names or coordinates, outside a region of interest or from distrustful sources are rarely suitable for biodiversity analyses. We can use the column .summary to filter valid records passing in all tests (i.e., flagged as “TRUE”). Next, we use the bdc_filter_out_falgs function to remove all tests’ columns (that is, those starting with “.”).
output <- check_pf %>% dplyr::filter(.summary == TRUE) %>% bdc_filter_out_flags(data = ., col_to_remove = "all") #> bdc_fiter_out_flags: #> The following columns were removed from the database: #> .scientificName_empty, .coordinates_empty, .coordinates_outOfRange, .basisOfRecords_notStandard, .coordinates_country_inconsistent, .summary
You can use qs::qsave() instead of write_csv to save a large database in a compressed format.