Data collection and input - nicetea02/Dashboard GitHub Wiki
Script in python for data collection from file structure and writing to DB.
For every parameter file of the VeriSeq NIPT, one one python script for collection and writing is present:
The data collection scripts need to be executed in a specific order due to the restrictions regarding primary keys in the MySQL DB.
Every script is based on a similar structure.
Duplicate detection
Firstly, a connection with the MySQL database and a list is created containing the primary key column of the table. This list is compared to a list containing all folder names of the NIPT_Output folder. When a non-match is found, the record will be collected and created.
Data collection
The next part of the script is the navigation to the correct directory containing the correct file. This file structure is always the same and different parameters will be found in different files. A list containing the parameters collected from the file is created.
Data insertion
Finally, the list described above will be used as values inside a SQL query in order to insert the parameters into the database.