meeting 2024 09 20 n57 - JacobPilawa/TriaxSchwarzschild_wiki_5 GitHub Wiki
Context
Plots
- First, here's an updated plot of the 1d panels for the fiducial case with all the scalings included. Note that the best selected scales are plotted as very faint and small black points, which overwhelmingly correspond to the scale = 1.005 case.
- After plotting this, I noticed that a lot of the black points showing the "best scale" models aligned with the scale = 1.005 case (the actual number if roughly ~80% of the best scaled models are from s=1.005 for the NNLS, no dummies, no outer 4 case). I remembered that last week, the "best scale" cases that I was looking at in fact excluded the s = 1.005 case since the choice of scalings had been instead optimized for the kinem, all bins, all dummies case.
- I have some more diagnostics, but this fact clued me off in the first place that the error was simply using the wrong subset of scales for the "best scaled" GPR and dynesty runs. I'll come back to this later
| Plot |
|---|
| images/240920/panels_nnls_no_outer_4_no_dummies.png |
- We also wanted to see a direct comparison of the NNLS and Kinem chi2 with everything included compared to what our "fiducial" case is:
- Note that the points in these plots are the "best scales" models rather than the scale = 1 cases we've seen before.
- I think what we see here is consistent with what we expect -- the kinem has the slight "bend" which ultimately comes from the chi2 associated with the outer bins, whereas this isn't as huge of a problem with the NNLS models. The NNLS models track each other quite well as we saw with the scale = 1.0 case.
- Given that the two NNLS cases track each other so well, it's still quite surprising the cornerplots are so discrepent, but there's some additional work on that front below.
- I also took a look at which of the scalings end up contributing to the "best scalings," and for the fiducial case of (NNLS, No Outer 4, No Dummies), scale = 1.005 contributes ~80% of the final models. Strangely enough, unless I'm doing something wrong, none of the scale = 1.0 models get selected, and only a few of the 0.995 models and a few of the higher scales get selected. Essentially it just seems like our "best scaled" models are coming from more or less a of the scalings.
- This clued me off to the idea that maybe the scalings I am including in our fiducial cornerplot (which were optimized for the kinem, all moments, all bins case) might need to be adjusted, and that does seem to be the case given the results below.
- Again, the fact that ~80% of the models are coming from a single scaling (s = 1.005) clued me off to the scalings being a strange issue, especially given that these plots below still track each other quite well (in the NNLS case).
| Kinem | NNLS |
|---|---|
| [images/240920/kinem_comp.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240920/nnls_comp.png) |
Cornerplots
-
Now for the main updates -- I think I was able to moderately tame the issue we were dealing with last time, where our "fiducial" case (NNLS, No Outer 4, No Dummies) cornerplot was very discrepent than the other 4 cases.
- I tamed this issue by realizing that last week, when we were looking at the "best scale" results, we had only been including the (0.99, 0.99, and 1.00) scalings. I had chosen these three scalings because expanding the scalings beyond this led to parameters being shoved to the boundary. Crucially, however, this obsevation was made looking at the Kinem, No Dummies, All Bins case, which has a slightly different preferred minimum than our new fiducial case using NNLS, no outer 4, no dummies.
- The new fiducial case (again, NNLS without the outer 4 bins and without dummies) seems to have its lowest chi2 models closer to the 1.005 scaling rather than the 0.995 case.
- I re-ran GPR and dynesty on a few cases below now with a better aligned set of scaled models. When I use the (1.0, 1.005, and 1.01) set of scalings, it really seems like our issue is greatly improved and more in line with the results of the other cases.
-
Lingering issue:
- One thing I'm still trying to understand is why further expanding the pool of available models beyond (1.0, 1.005, and 1.01) seems to bring the issue back. There are very few models which get selected when adding these additional scalings, so I hadn't anticipated the answer to appreciably change.
- My current hypothesis: when trying to use all the scalings beyond (1.0, 1.005, and 1.01), a few well fitting models get picked up from the larger/smaller scales. I expect we're running into an issue where our fits are being dominated by one or two really well fitting points, and thus GPR and dynesty is extrapolating off the grid to accomodate one or two really wel fitting points. I'm following up on this now though and hope to have some answers soon!
| Description | Plot |
|---|---|
| NNLS No Dummies, No Outer 4, Include All Scalings and Select Best | images/240920/nnls_best_scales_no_outer_4_no_dummies_all_scalings_grid_alpha_K60_nu1.5-1.png |
| NNLS No Dummies, No Outer 4, Include (1.0, 1.005, 1.01, 1.02) and Select Best | images/240920/nnls_best_scales_no_outer_4_no_dummies_1p00_1p005_1p01_1p02_grid_alpha_K60_nu1.5-1.png |
| NNLS No Dummies, No Outer 4, Include (1.0, 1.005, 1.01) and Select Best | images/240920/nnls_best_scales_no_outer_4_no_dummies_1p00_1p005_1p01_grid_alpha_K60_nu1.5-1.png |
| NNLS No Dummies, No Outer 4, Include (0.995,1.0,1.005,1.01) and Select Best | images/240920/nnls_best_scales_no_outer_4_no_dummies_0p995_1p00_1p005_1p01_grid_alpha_K60_nu1.5-1.png |
| From Last Time: NNLS No Dummies, No Outer 4, Include (0.99, 0.995, 1.00) and Select Best | images/240912/nnls_best_scales_no_outer_4_no_dummies_grid_alpha_K60_nu1.5-1.png |
Follow Up Information
- One quick sanity check -- here are histograms of the chi2 for the cases above where you can see the preference for the s=0.995 for kinem and s=1.005 for the NNLS models.
| NNLS, No Dummies, No Outer 4 | NNLS, No Dummies, All Bins | Kinem, No Dummies, No Outer 4 | Kinem, No Dummies, All Bins |
|---|---|---|---|
| [images/240924/nnls_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_all_bins_no_dummies.png) | [images/240924/kinem_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_all_bins_no_dummies.png) |
- A quick comparison of the different cases we have plotted against one another. The blue points are the full sets of models in all grids, whereas the orange points are the "best selected" models. Note I am including all scalings here.
| Less Zoomed | More Zoomed |
|---|---|
| [images/240924/comparisons.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/comparisons_zoom.png) |
- I'm adding some additional plots and diagnostics here following our email exhanges related to the information above. First, here are the 1d chi2 vs. parameters for all scalings and the resulting "best scales" from each. Note that in these plots I'm including everything. I've also reduced the legend so that we can see all relevant points. In the second row, you can see the histogram for which models get picked when selecting the "best scales" version for each case:
| NNLS No Dummies, No Outer 4 | NNLS No Dummies, With Outer 4 | Kinem No Dummies, No Outer 4 | Kinem No Dummies, With Outer 4 | |
|---|---|---|---|---|
| Plot | [images/240924/panels_nnls_no_outer_4_no_dummies.png]] ](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/panels_nnls_all_bins_no_dummies.png) | [images/240924/panels_kinem_no_outer_4_no_dummies.png]] ](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/panels_kinem_all_bins_no_dummies.png) | ||
| Hist | [images/240924/hist_nnls_no_outer_4_no_dummies.png]] ](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/hist_nnls_all_bins_no_dummies.png) | [images/240924/hist_kinem_no_outer_4_no_dummies.png]] ](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/hist_kinem_all_bins_no_dummies.png) |
- And here are the individual scalings plotted individually:
| Case | Scale=0.97 | Scale=0.99 | Scale=0.995 | Scale=1.0 | Scale=1.005 | Scale=1.01 | Scale=1.02 | Scale=1.03 |
|---|---|---|---|---|---|---|---|---|
| NNLS No Dummies, No Outer 4 | [images/240924/nnls_scale_0p97_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_0p99_no_outer_4_no_dummies.png) | [images/240924/nnls_scale_0p995_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p00_no_outer_4_no_dummies.png) | [images/240924/nnls_scale_1p005_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p01_no_outer_4_no_dummies.png) | [images/240924/nnls_scale_1p02_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p03_no_outer_4_no_dummies.png) | ||||
| NNLS No Dummies, With Outer 4 | [images/240924/nnls_scale_0p97_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_0p99_all_bins_no_dummies.png) | [images/240924/nnls_scale_0p995_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p00_all_bins_no_dummies.png) | [images/240924/nnls_scale_1p005_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p01_all_bins_no_dummies.png) | [images/240924/nnls_scale_1p02_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/nnls_scale_1p03_all_bins_no_dummies.png) | ||||
| Kinem No Dummies, No Outer 4 | [images/240924/kinem_scale_0p97_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_0p99_no_outer_4_no_dummies.png) | [images/240924/kinem_scale_0p995_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p00_no_outer_4_no_dummies.png) | [images/240924/kinem_scale_1p005_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p01_no_outer_4_no_dummies.png) | [images/240924/kinem_scale_1p02_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p03_no_outer_4_no_dummies.png) | ||||
| Kinem No Dummies, With Outer 4 | [images/240924/kinem_scale_0p97_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_0p99_all_bins_no_dummies.png) | [images/240924/kinem_scale_0p995_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p00_all_bins_no_dummies.png) | [images/240924/kinem_scale_1p005_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p01_all_bins_no_dummies.png) | [images/240924/kinem_scale_1p02_all_bins_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/kinem_scale_1p03_all_bins_no_dummies.png) |
Adding a the last minute
- I also recreated the 1d panels as I have above but limiting to only the three central scales which seem to work for the fiducial case. I just wanted a quick sanity check that the s = 1.005 points are indeed preferentially the lowest, and this seems to be the case:
| s=1.0 | s=1.005 | s=1.01 | Best Scales |
|---|---|---|---|
| [images/240924/trimmed_nnls_scale_1p00_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/trimmed_nnls_scale_1p005_no_outer_4_no_dummies.png) | [images/240924/trimmed_nnls_scale_1p01_no_outer_4_no_dummies.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/240924/trimmed_panels_kinem_all_bins_no_dummies.png) |