Specific instructions for NAE - jniedzie/SVJanalysis_wiki GitHub Wiki

Several bash scripts have been written to perform NAE training, evaluation and making plots. This page describes them in the order they should be executed.

Training, evaluation and base plot

The script run_nae_training_evaluation_and_plots_example.sh automatically performs the NAE training and evaluation, and makes a series of default plots, for the selected trainings and configs. Make a copy of this example script and edit it to your needs.

In this script, you need to update (see comments in the script):

  • The name of the config for which to run the NAE
  • The training model numbers to run

Important note: If you send this script twice without changing anything, you will overwrite the previous trainings already performed! If you want to send more trainings, you have to increase the training model numbers!

Important note 2: You can perform the evaluation a second time, changing the background and/or signal samples following this procedure:

  • comment out the line performing the training: echo "python train.py -c ${cfg} -v $i" >> ${runner}
  • change the plots_path variable in the config
  • change background and/or signal samples in the config
  • run the script again

The evaluation on the new samples and defaults plots for these new samples will be made and stored in a new folder (defined by plots_path). Do not forget to uncomment the training when you send new trainings :wink: .

Producing best epoch information

The script produce_best_epoch_info_example.sh determines the best epoch number of the NAE trainings and writes this information along with others in a file, for the selected trainings and configs. Make a copy of this example script and edit it to your needs.

You should check the plots produced to find the best epoch to verify if the point of EMD increase was correctly found. In particular, you should adapt the value of the -avg flag. This depends on the number of parameter updates per epoch.

Making AUC tables

The script run_auc_tables_example.sh produces AUC tables for the selected trainings and configs. Make a copy of this example script and edit it to your needs.

Different kind of AUC tables can be made:

  • AUC tables for all training checkpoints
  • AUC tables for the best epoch
  • Average AUC tables for an ensemble of trainings at their best epoch

All command-lines, with comments, to perform these plots are provided in the script.