PyTrousse functional requirements - HK3-Lab-Team/pytrousse GitHub Wiki

PyTrousse functional requirements

1. Data loading/saving

  1. The system must be able to load raw data from a CSV file / pandas DataFrame.
  2. The system must be able to save to disk the data with the associated preprocessing operations that have been performed on that data.
  3. The system must be able to read the preprocessed data from disk along with the previously performed preprocessing operations. (NOTE: See how sklearn does this part because if we save the info as json, we should also write code to validate the schema. Try to avoid this)

2. Columns handling

  1. The system must be able to distinguish between metadata columns and feature columns.
  2. The system must be able to categorize columns depending on their type, i.e. numerical columns, categorical columns, boolean columns, string columns, mixed columns.
  3. The system must be able to identify columns that contain a lot of missing values (more than 99.9% of the total number of samples by default).
  4. The system must be able to identify columns that contain the very same value.
  5. The system must be able to identify columns with the same name.

3. Preprocessing

  1. The system must be able to store the preprocessing operations performed on the data, with a reference to the name of the column(s) the operation has been performed on, the name of the resulting column(s), the type of operation performed. In case encoding is performed, the system must also store the type of encoding used and a map from the encoded values to the original values.
  2. The system must be able to retrieve preprocessing operations that match user-defined requirements, i.e. performed on a specific column, which type of operation has been performed, which column has been derived from it, which type of encoding has been used (if applicable).
  3. Encoding
    1. The system must be able to perform data binning on a numerical column, based on the binning threshold values provided by the user, and to create a new column with the corresponding bin indexes. The threshold values can be defined for sample sub-groups, where the grouping is performed based on the value of another column.
    2. The system must be able to perform the One Hot encoding of a categorical column and to create a new column with the encoded values.
    3. The system must be able to perform the Ordinal encoding of a categorical column and to create a new column with the encoded values.
    4. The system must be able to perform the Ordinal encoding of multiple categorical columns by indexing the combinations of their unique values and to create a new column with the corresponding indexes.
  4. Correction
    1. The system must be able to replace strings (e.g. "---", "ASSENTI", "NV") or substrings (e.g. "°", ",") in mixed type columns with strings provided by the user.
    2. The system must be able to replace string values in mixed type columns that represent out-of-scale numerical values (e.g. “>800”) with an appropriate numerical value (e.g 880).
    3. After correction, the system must be able to recompute the type of the columns, as defined in 2b
    4. The system must be able to perform the Ordinal encoding of multiple categorical columns by indexing the combinations of their unique values and to create a new column with the corresponding indexes.
  5. Anonymize
    1. The user must define the columns which contain private information.
    2. The system must be able to associate a unique ID to the different combinations of information contained in the private columns.
    3. The system must be able to replace the private columns with a column containing the corresponding unique ID.
    4. The system must store the association between private information and the corresponding unique ID in a separate file.
⚠️ **GitHub.com Fallback** ⚠️