Data Samples Preparation - PanDAWMS/InVEx GitHub Wiki

Each data sample uploaded on server and stored in RAM on server-side must be preprocessed and prepared to be transferred to clients.

Data Preparation is implemented in /core/form_reactions.py.

Procedures csv_file_from_server and new_csv_file_upload provide the following actions:

• open data file
• read file
• clean data file from NaNs
• split data file into numeric data (numeric_dataset) and auxiliary data (auxiliary_dataset). Numeric data will be used in clusterization and auxiliary (not numeric) data - will be stored separately and used for data categorization.
• numeric data is normalized - norm_dataset
• calculated statistics of the normalized dataset
• all these datasets are saved in history file with the unique ID