Home - olievortex/olieigra GitHub Wiki
What is olieigra
Olieigra is a package that allows an application to scan for and ingest IGRA2 data as a stream. This gives the calling application complete control over the transformation of the data and do so without consuming vast amounts of memory.
Olieigra scans a folder for IGRA2 files. If it finds a .zip file, it will scan through it also looking for IGRA2 files. It does this without expanding the zip file.
Walkthroughs
Callbacks
The olieigra package interacts with your application via callbacks. The following callbacks are used:
def start_file(self, filename: str) -> bool: Called whenever the crawler discovers a file within the folder, or within a zip archive. Returning True will cause olieigra to process the file. Returning False will cause it to skip the file. You should implement your destination initiation code here.
def finish_file(self, headers: int, rows: int): Called when the processing of a file is complete. The number of headers and rows processed are passed for informational purposes. You should implement your destination clean-up code here.
def parse_header(self, header: olieigra.HeaderModel) -> bool: Called when a header record is parsed. Returning a True will cause olieigra to process the associated body records. A False will skip the body records. This allows you to filter out records you are not interested in. The HeaderModel object contains all the values read from the file.
def parse_body(self, body: list[olieigra.BodyModel]): Called when all the body records have been processed related the prior parse_header callback. The body parameter is a list of olieigra.BodyModel objects that contain all the values read from the file.