Chapter 3 audioBlastIngest

The ingest service is responsible for bringing data about recordings, deployments, annotation, taxa and traits into the audioBlast system. Data sources are specified as modules within the audioBlast API, which give details of each data source and the processing that is expected to be performed by the ingest process. The ingest process itself is managed on audioBlast infrastructure by the audioBlastIngest package. This package obtains data from the sources specified in the API modules, performs and requested transformations, and loads the data into the audioBlast database.

Note: The database generates analysis tasks using triggers as data is inserted or updated, however the analyses are performed by a separate processes to the ingest.

Updates are handled periodically (generally nightly) by the ingest server. Creating an ingest task is a simple three step process.

  1. Third party publishes data online.

  2. audioBlast team work with third party to create an API module to map the published data on to audioBlast data structures.

  3. The audioBlastIngest package handles the import of the data.

3.1 Source modules

3.1.1 Fields

3.1.2 Mapping columns

3.1.2.1 Optional processing done by audioBlast

3.1.2.1.1 Specifying processors
3.1.2.1.1.1 SourceR: Automatically prepend a source column
3.1.2.1.1.2 date2dateAnTime(): Date and time mapping

You can map datetime fields to Date and use the date2dateAndTime() processor to split them into separate date and time fields.