Fitter_Validation_NuMu_sample - PetrilloAtWork/sbncode GitHub Wiki

h1. Fitter Validation NuMu sample

Last Updated: 07/05/2019

h2. Which version Should You Use Right Now?

Version 2.2 has the most up-to-date configuration in all three detectors.

h2. Change-log

  • Version 1.0: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/samples *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/processed ** sbncode version: v8.03.00
  • Version 1.1: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/samples *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/processed_1.1 ** sbncode version: v8.12.00 ** Changes: *** Tweaks to fiducial volume in all three detectors *** Don’t apply bnbcorrection to calculate “weight” of SBND events. The flux is already valid in the sample, so there is no need to apply the bnbcorrection. *** Remove some SBND sample files from processing which are duplicates of each other. *** Updated output format in sbncode (now includes POT calculation)
  • Version 1.temp-wgt: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/samples *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/processed_1.temp_wgh ** sbncode version: v08_12_00-tempwgh ** Changes: *** Update to sbncode output format to flatten weights *** Contains bugfix in “reco_energy” calculation.
  • Version 2.a: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight/samples *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight/processed_2.a ** sbncode version: v8.12.00 ** Changes: *** New sample with approximately 20% of simulation re-processed in each detector *** Numerous changes to systematic weights. Switch from 100 to 1000 multisims. *** Notes for processing: **** New sample files take the same generation information from the old sample and split them into 10 files. Each new file contains the subrun (including POT) information of the original file. Thus, when directly looking at the sample files, one must divide the POT by 10 to get the accurate number. The processed files already have this accounted for. **** The weights for non-resonant backgrounds are split into two -- one for CC events and one for NC events. When building (e.g.) a covariance matrix, one should apply one of the weights to all CC events, and the other to all NC events. This will in effect cause non-resonant background systematic uncertainties to be calculated separately for CC and NC, which was the method used in the SBN Proposal. (This is not the method implemented currently in EventWeight).
  • Version 2.0: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight2/samples/ *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight2/processed_2_0/ ** sbncode version: v08_12_00-tempwgh ** Changes: *** More fixes in “reco_energy” calculation *** There are now 1000 systematic variations in cross section uncertainties. This leads to a much better match between the reproduced uncertainties and what was in the proposal. *** NOTE: currently, only uBooNE and ICARUS is available in this version. SBND is awaiting processing. *** Notes for processing (same as sample 2.a): **** New sample files take the same generation information from the old sample and split them into 10 files. Each new file contains the subrun (including POT) information of the original file. Thus, when directly looking at the sample files, one must divide the POT by 10 to get the accurate number. The processed files already have this accounted for. **** The weights for non-resonant backgrounds are split into two -- one for CC events and one for NC events. When building a covariance matrix, one should apply one of the weights to all CC events, and the other to all NC events. This will in effect cause non-resonant background systematic uncertainties to be calculated separately for CC and NC, which was the method used in the SBN Proposal. (This is not the method implemented currently in EventWeight).
  • Version 2.2: ** location: *** samples: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight2/samples/ *** processed: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight2/processed_2.2/ ** sbncode version: see branch feature/mastbaum_event ** Changes: *** More more fixes in “reco_energy” calculation *** Now includes sample in SBND with the correct updated flux -- 110m along baseline and offset 0.457m along x. Events in SBND are scaled up by 1.21 (in the “Proposal” configuration only) to partially account for the difference between this and the proposal position (as was done in previous samples). *** Includes scaling of CC events by 0.9813 and NC by 1.0515 to account for changes to Genie event generation since proposal (“Proposal” configuration only) *** Includes by-hand scaling of reco_energy by 1.031 to account for apparent difference between current reco_energy calculation and the one used for the proposal (“Proposal” configuration only) *** Modern sample has updated length calculation as detailed in SBN docdb 13300-v4. *** Now includes “MiniBooNE” flux weights -- these are systematic variations generated with the exact same random numbers as in the old MiniBooNE framework (which was used to generate flux uncertainties for the proposal).

h2. Sample Definitions

This page documents a set of inclusive muon neutrino samples used to benchmark different fitter software reproducing the SBN oscillation sensitivity.

The processed samples (produced in the sbncode framework) exist at /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/processed. There are two sets of three files (one for each detector):

  1. Proposal Sample
  • output_SBNOsc_NumuSelection_Proposal_Icarus.root
  • output_SBNOsc_NumuSelection_Proposal_SBND.root
  • output_SBNOsc_NumuSelection_Proposal_Uboone.root
  1. Modern Sample
  • output_SBNOsc_NumuSelection_Modern_Icarus.root
  • output_SBNOsc_NumuSelection_Modern_SBND.root
  • output_SBNOsc_NumuSelection_Modern_Uboone.root

The “Proposal Sample” represents the sample as of the SBN proposal. It removes all MEC events and has a reduced SBND fiducial definition. The “Modern Sample” represents samples with the latest genie generation. See below for the precise differences.

Both samples are generated from the same monte carlo, which exists at: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/samples/ (version 1) and /pnfs/sbnd/persistent/users/gputnam/numu_simulation_reweight2/samples/ (version 2).

Both samples have the same POT in each detector. The value of the POT in each detector can be found in the file: /pnfs/sbnd/persistent/users/gputnam/numu_simulation_12_05_2018/processed/DETECTOR_POT where DETECTOR is one of SBND, ICARUS, UBOONE.

h3. Differences between the Proposal and Modern configurations

Modern:

  • Uses genie generation mode “Default+CCMEC+NCMEC”
  • Places SBND at 110m from the beam
  • Uses SBND size of 4x4x5m

Proposal:

  • Uses genie generation mode “Default+CCMEC+NCMEC” but selection excludes all events with a “kMEC” genie interaction mode
  • Scales SBND events by (110 / 100)**2 to effectively place SBND at 100m from the beam (still ~5% normalization difference due to different location of beam at SBND in modern generation)
  • Uses SBND size of 4x4x3.65m

The differences in the Proposal sample are there to faithfully reproduce the selection in the SBN Proposal as best as possible.

h2. Sample Contents

All samples files contain a number of histograms (for personal use) and a single TTree -- sbnana. The “sbnana” tree contains two branches: “events” and “numu_interaction”. Again, the “numu_interaction” branch is intended for personal use. All fitters should only need to depend on the “events” branch. Each index in the “events” branch contain a vector of Event objects as documented at: https://cdcvs.fnal.gov/redmine/projects/sbncode/repository/revisions/master/entry/sbnanalysis/core/Event.hh. Note that different versions of the sample have different versions of the event definition. You should look at the event definition that corresponds to the sbncode version of that sample.

Each sample is filled with a mix of (signal) CC numu interactions and (background) NC numu interactions. The selection steps are as follows:

  • Start with a sample of all muon neutrino interactions in each detector
  • Cut out all interactions outside the fiducial volume (using the truth interaction vertex)
  • Cut out all interactions that do not produce a muon or pion track (using the truth PDGID)
  • Calculate the length of all tracks coming from the truth vertex inside the active volume (using truth tracks)
  • Cut out all contained tracks with length < 50cm and all exiting tracks with length < 100cm
  • (Proposal Sample only) cut out all events with the “kMEC” interaction (at truth level)

The active and fiducial definitions are defined in the attached fhicl file NumuConfig.fcl.

Each interaction has a weight that is a product of:

  • the selection efficiency (set at 80% as in the proposal)
  • the bnbcorrection weight (applied only in SBND and ICARUS).
  • scaling up events in SBND by 1.21 to “move” the detector from 110m (current flux config) to 100m (proposal flux config) [Proposal sample only]
  • (starting in sample version 2.2) scaling CC and NC events to account for changes to Genie since the proposal [Proposal sample only]

This weight is contained in the RecoInteraction object, and should be applied by fitters.

Each interaction has a reconstructed energy (“reco_energy” in the RecoInteraction) which is intended to be used as the main observable for fitters. The reconstructed energy is calculated from truth information by summing up the smeared energy of all charged tracks emanating from the interaction vertex. The muon track energy is smeared using the function -Alog(BL) w/ A = 0.102, B=0.000612 1/cm (leaving) or 2% (contained). All other tracks have their energy smeared by 5%. Only charged tracks with a kinetic energy of at least 21MeV have their energies included in the reconstructed energy.

Each interaction also has a list of weights generated by EventWeight to be used for systematic variations contained in the Interaction object. The list of weights with the configurations are specified in the attached file “sbn_eventweight.fcl” (which was used to generate these weights).

h2. Which Systematic Variations to Use

There are a number of weights generated in the sample to implement systematic variations in order to calculate uncertainties. Not all of the generated uncertainties should be used to replicate the proposal. Only the following should be used on every event:

h3. Flux:

  • expskin_FluxUnisim
  • horncurrent_FluxUnisim
  • kminus_PrimaryHadronNormalization
  • kplus_PrimaryHadronFeynmanScaling
  • kzero_PrimaryHadronSanfordWang
  • nucleoninexsec_FluxUnisim
  • nucleonqexsec_FluxUnisim
  • nucleontotxsec_FluxUnisim
  • piminus_PrimaryHadronSWCentralSplineVariation
  • pioninexsec_FluxUnisim
  • pionqexsec_FluxUnisim
  • piontotxsec_FluxUnisim
  • piplus_PrimaryHadronSWCentralSplineVariation

h3. Interaction:

  • genie_ccresAxial_Genie
  • genie_ncresAxial_Genie
  • genie_qema_Genie
  • genie_NC_Genie

In addition, the following weights should be applied to all CC events:

  • genie_NonResRvbarp1pi_Genie
  • genie_NonResRvbarp2pi_Genie
  • genie_NonResRvp1pi_Genie
  • genie_NonResRvp2pi_Genie

And, the following weights should be applied to all NC events:

  • genie_NonResRvbarp1piAlt_Genie
  • genie_NonResRvbarp2piAlt_Genie
  • genie_NonResRvp1piAlt_Genie
  • genie_NonResRvp2piAlt_Genie

This separation between CC and NC events is to work-around a detail of the EventWeight framework for the purposes of reproducing the proposal. In the proposal, the “NonRes” weights were thrown separately for CC and NC interactions. However, in the EventWeight framework the “NonRes” weights are thrown together. Separating out the weights into two copies (each with a different random seed) sets the weights as they were in the proposal.

In practice though, the effect of throwing CC/NC weights together and throwing them separately for “NonRes” seems to be small.

The other variations in the output files are there to study the effect of other uncertainties from Genie that were not included in the proposal. The “bnbcorrection” weight is there to fix a bug in the flux files used to generate events in MicroBooNE and ICARUS (but not SBND) in the sample. However, the “bnbcorrection” is already applied in the “RecoInteraction::weight” variable. Thus, fitters should not account for it.

Starting in sample version 2.2, there are also flux weights generated in the “MiniBooNE” configuration. These are the same as the normal flux weights, except with the same random numbers as were used in the old MiniBooNE framework. This is the framework which was used to generate the proposal flux uncertainties, so for the purposes of comparing to the proposal it may be desirable to use these instead of the normal flux weights.

h2. Steps for Processing

Notes:

  • there is an anomalous event in the SBND sample that has some systematic variations from EventWeight with a weight of >100,000. It is recommended that fitters remove the event as an outlier.
  • in the RecoInteraction object, the “weight” variable should be applied as a weight to each event
  • also in the RecoInteraction object, the “reco_energy” variable is intended to be the main observable used by fitters (and is the one which was used for the SBN proposal)
  • Histograms of specta for both the “Modern” and “Proposal” samples are attached to this wiki for reference/comparison ** The histograms have a scaled POT of 6.6e20 for SBND and ICARUS and 1.32e21 for MicroBooNE ** The bins for all histograms are: [0.2, 0.3, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1., 1.25, 1.5, 2., 2.5, 3.]