Evaluation and plotting

The best model so far for the jet-like gun on the 0.5 dr eval seems to be: /eos/user/m/mgarciam/datasets_mlpf/models_trained_CLD/gun_drlog_v9/.
I trained another model with dr 01 but the email seems very similar.

The evaluation ran for the note for the dr 0.5 dataset (jet-like gun): python -m src.train_lightning1 --data-test /eos/experiment/fcc/users/m/mgarciam/mlpf/CLD/eval/gun_dr_05_v0_091224/pf_tree_{1..100}.root --data-config config_files/config_hits_track_v4.yaml -clust -clust_dim 3 --network-config src/models/wrapper/example_mode_gatr_e.py --model-prefix /eos/user/m/mgarciam/datasets_mlpf/models_trained_CLD/eval_comp_05/ --wandb-displayname eval_gun_drlog --num-workers 0 --gpus 1 --batch-size 10 --start-lr 1e-3 --num-epochs 100 --optimizer ranger --fetch-step 0.1 --condensation --log-wandb --wandb-projectname mlpf_debug_eval --wandb-entity fcc_ml --frac_cluster_loss 0 --qmin 1 --use-average-cc-pos 0.99 --lr-scheduler reduceplateau --tracks --correction --ec-model gatr-neutrals --regress-pos --add-track-chis --load-model-weights /eos/user/g/gkrzmanc/results/2024/E_PID_02122024_dr05_s6500_3layer_pid_GTClusters_all_classes_PID/_epoch=0_step=4500.ckpt --freeze-clustering --predict --regress-unit-p --PID-4-class --n-layers-PID-head 3 --separate-PID-GATr

% This is from Gregor and I have not tested it Simplified evaluation and plotting script:

python scripts/auto_plot_results.py --datasets dr_050 --gpu 0 --n-files 10 --path /eos/user/path/to/training

The script will look for new ckpt files in the training directory and create a subfolder eval_results with evaluation results and plots for each ckpt. In each model folder (e.g. epoch0step500), the files eval_done.txt and plotting_done.txt are created after each step is completed. In order to redo a step (e.g. plot again), just remove the corresponding txt file. The plotting commands are inside the script and maybe need to be modified for a specific case, i.e. if you need to include --use-gt-clusters or if we change the model architecture. Probably the python executable needs to be modified as well.

Plots

Plotting code

The main plotting code is in src/evaluation/refactor/plot_results.py. A typical plotting command will look like this: python src/evaluation/refactor/plot_results.py --path {path} --preprocess class_correction --output_dir filt_LE_CH_400bins_epsilon005 --mass-only

Preprocessing

The eval command generates a dataframe containing info about matched particles for ML and Pandora. Sometimes, it is useful to do quick analyses by, for example, only considering certain particles, or removing clusters that don't have good condensation points, etc. For this reason, we have a preprocessing script at preprocess.py. What preprocessing filtering techniques to apply is specified in the --preprocess flag of the plotting command above.

List of important plots

Keeping track of why each plot was made, what bugs were fixed etc.

The plots are divided into two folders, summary_plots and individual_plots. Typically the important plots are in summary_plots.

• E_resolution_per_PID_full_comparison.pdf: Comparison of reco (left) and energy correction (right) plots for different PIDs
• overview_Efficiency_FakeRate_ConfusionMatrix.pdf: Overview of the confusion matrix by energy, as well as efficiency, fake rate and fake energy rate for each PID category. The main summary plots are described below:
• mass_resolution.pdf: the main result - mass (left) and energy (right) resolution histogram
• mass_breakdown_by_PID.pdf: for each PID, see the contribution for event energy. This doesn't include fakes, that's why we added mass_breakdown_by_category.pdf
• E_breakdown_by_PID_category.pdf shows each PID category (e/CH/NH/gamma) separately. E_breakdown_by_PID_category_X_Y_GeV.pdf shows the same plots in energy slices from X to Y GeV.
• E_breakdown_by_PID.pdf is similar to E_breakdown_by_PID_category, except here the results are per matched PID.
• phi.pdf and theta.pdf give distributions of the angular differences for different particles.
• track_cluster_eval.pdf contains histograms with track-cluster assignation efficiency and similar metrics.

Hss dataset plots as of November 2024

Plots for Hss dataset (epoch 4 clustering) are [here] - the plots also include comparison with ML GTC in folder comparison (https://cernbox.cern.ch/files/spaces/eos/user/g/gkrzmanc/results/2024/test_1311_additional_features_150files_model_epoch_4_step_50000)

Plots for Hss dataset (epoch 5 clustering) are here - these include some smaller 'experiments' i.e. removing photons etc.

Plots for Hss dataset with GT clustering and particle gun EC: here

Plots for Hss dataset with model clustering epoch 4 and particle gun EC: here

Particle gun dataset plots

TODO