meeting 2024 10 01 n57 - JacobPilawa/TriaxSchwarzschild_wiki_5 GitHub Wiki
Context
- We're in the final stages of wrapping up N57, and I think the points here address many of the open questions we had last time. Specifically:
- I've produced two diagnostics plots for N57 that I'm not sure change the current picture by any means but they're useful to see. These plots are the histogram of scale factors in bins of chi2, and the 2d panels showing our models vs. parameters colored by chi2. We wanted to see the former plot in the meeting last week to see if certain chi2 bins had preferences for certain scale factors. To me, it seems like generally speaking, all of the chi2 bins seem to include more models from higher scale factors than lower scale factors which I thikn is a bit surprising to me. I think I would have expected poorly fitting models to come from a more diverse array of scale factors, but maybe this is really just an artifact of the fact that our masses are preferentially higher relative to the original grid of models (since the points were seeded based on the Kinem chi2 (incorrect binning scheme, all bins, with dummies).
- I've run the jacknife test on the fiducial case of models (NNLS chi2, no dummies, no outer bins), and the 10 realizations I produced are in excellent agreement, suggesting that our minima are not determined by any points in particular. For this test, I've thrown out half the "best scaled" points in each iteration. I've run the GPR and dynesty with the same sets of parameters (K=60,nu=0.5). Note that the cornerplots are still showing the 3sigma values in the titles, but the vertical plot I produced compares the 1sigma values which are in excellent agreement for all parameters.
- We also wanted to look at the linking lengths favored by our GPR fits, and I think if we were to push further on anything, this would be where we push. Last time, we noticed that the nu = 0.5 GPR fits led to substantially better looking posteriors compared to the nu=1.5 case, and we hypothesized that something strange might be happening with the linking lengths. This does seem to be the case, but I'm really not sure how much our answers will change or if this is something to be concerned about.
- As a quick reminder, we're currently bounding the length scales to be [1e-2, 10.0]. The resulting fits for the nu=1.5 models seem to have the intermediate values for the linking lengths that Emily had hypothesized in our meeting last time. The nu=0.5 GPR models, on the other hand, consistently favor a linking length = 10 (or very very close to the boundary).
- I ran a quick test where I allow the linking length to go up to 100, and this appears to alleviate the linking length issue with 0 change at all to the parameters. For this reason, I'm not particularly worried about the linking lengths favoring the boundary, but happy to do additional testing on the linking lengths and relationship to nu if we see fit.
Plots
- Final cornerplot, updated:
| Plot |
|---|
| images/241001/test-1.png |
- First, here is the quick jacknife test we wanted done. To produce this, I've run 10 jacknife iterations in which I randomly throw out 50% of the models in the fit, and then re-fit with GPR and dynesty. These GPR and dynesty runs were all produced with the same settings (K = 60, nu = 0.5), and the results are remarkably consistent. The vertical plots show the 1sigma regions, and I've included the cornerplots for the 10 cases (which have the 3 sigma errors in the titles still).
| Jacknife Results |
|---|
| images/241001/vertical_comp.png |
| R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 |
|---|---|---|---|---|---|---|---|---|---|
| [images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_0_grid_alpha_K60_nu0.5-1.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_1_grid_alpha_K60_nu0.5-1.png) | [images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_2_grid_alpha_K60_nu0.5-1.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_3_grid_alpha_K60_nu0.5-1.png) | [images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_4_grid_alpha_K60_nu0.5-1.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_5_grid_alpha_K60_nu0.5-1.png) | [images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_6_grid_alpha_K60_nu0.5-1.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_7_grid_alpha_K60_nu0.5-1.png) | [images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_8_grid_alpha_K60_nu0.5-1.png]]](/JacobPilawa/TriaxSchwarzschild_wiki_5/wiki/[[images/241001/241001_nnls_no_outer_4_no_dummies_jackknife_9_grid_alpha_K60_nu0.5-1.png) |
- I also created the plot we discussed last time in which I plot the scale factor distribution of for progressive cuts in chi2. Interestingly, it seems like generally speaking the higher scale factors models are always being selected in higher numbers than the low scale factors:
| Histogram binned by Chi2 |
|---|
| images/241001/chi2_histogram.png |
- Not something we directly talked about, but here are the 2d chi2 vs. parameter panels, with the lowest chi2 model marked by the red square. It doesn't seem that the lowest model is driving us toward any particular plcae in M/L (i was worried there may have been a bunch of points in some bizarre corner of the space not visible in the 1d panels, but that does not seem to be the case):
| 2D Panels |
|---|
| images/241001/2d_plot.png |
Linking Length Question
-
We also were wondering if the linking lengths had any role in the GPR and dynesty fitting, so here's some additional information on this front. I double checked and the my linking lengths are in fact bound by [1e-2, 10], and it actually appears that the good-looking nu=0.5 cases have linking lengths near the upper boundary of 10, whereas the linking lenghts associated with nu=1.5 are much more intermediate values for the lengths (which was Emily's hypothesis last time). I tried increasing the upper bound to 100, and the linking lengths mostly converge without changing the parameters at all.
-
Here are the linking lengths for the cases we were looking at last time (the order of linking lengths is BH, ML, Rho0, T, Tmaj, Tmin):
| K=30 | K=40 | K=50 | K=60 | |
|---|---|---|---|---|
| nu=0.5 | Matern(length_scale=[10, 10, 10, 10, 10, 10], nu=0.5) | Matern(length_scale=[10, 10, 10, 9.51, 10, 10], nu=0.5) | Matern(length_scale=[10, 10, 10, 10, 10, 10], nu=0.5) | Matern(length_scale=[10, 10, 10, 10, 10, 10], nu=0.5) |
| nu=1.5 | Matern(length_scale=[2, 3.65, 2.19, 1.57, 1.49, 1.38], nu=1.5) | Matern(length_scale=[2.39, 5.68, 2.18, 1.34, 1.92, 1.58], nu=1.5) | Matern(length_scale=[2.79, 9.28, 2.4, 1.45, 2.3, 1.59], nu=1.5) | Matern(length_scale=[3.13, 10, 2.78, 1.6, 2.2, 1.86], nu=1.5) |
- Here's the resulting cornerplot when I allow the linking lengths to go to 100. The linking lenghts here are: Matern(length_scale=[91.7, 100, 74.1, 42.6, 78.1, 60], nu=0.5)
| Linking Length Change |
|---|
| images/241001/TESTING_grid_alpha_K60_nu0.5-1.png |
