Summary: weekly mtg 20160601 (Byron, Matt, me) - mobeets/nullSpaceControl GitHub Wiki

Prep

A better cloud hypothesis

In the course of trying to choose the best tuning parameter for the cloud hypothesis, I realized that just taking the closest point in terms of row space (for both cloud and pruning) does just as good (pruning, except for on 20120709 when the covariance error gets slightly higher) or better (cloud) as the parameters I was using before. This made cloud a very competitive hypothesis for most dates.

Writing a paper

Just to review, two things I think needing resolving before we can start a paper are:

  1. Do our results change if we account for the monkey's estimate of the decoder? (Does using IME change the relative orderings of hypotheses?)

  2. Can we distinguish between hypotheses within the "task dependent" hypotheses? (Do the habitual, pruning, and cloud hypotheses make distinct predictions?)

1. Hypotheses with and without IME

See plots/allNew3/compare for comparisons of hypothesis errors in mean without ( copy.png) and with (.png) IME. As you can see, IME preserves the relative ordering for all dates except for 20120709, in which case under IME habitual is the best hypothesis, and then on 20120525 and 20120601 when cloud alone improves relative to the others. On all other dates, pruning is basically the best and unconstrained the worst on all sessions.

It's interesting to note that as far as mean shift goes, without IME, when you fit with thetaGrps and score with thetaActualGrps, this actually makes pruning as good as mean shift, which was not the case when we were scoring with thetaGrps as before (see issue #183). And adding in IME doesn't really change things; in some cases, e.g. for 2013 dates, mean shift might still be a tiny bit better than pruning or cloud.

2. Habitual vs. pruning vs. cloud

These hypotheses now seem to trade off in terms of who's the best. By date:

I've tried one thing to try to distinguish them (see issue #184). Basically, differences between the hypotheses are maybe 1/4 as large as the errors at predicting the observed values--so they're similar, but not identical. In fact, for most dates the more habitual differs from pruning, the worse habitual does. I've yet to try to tease apart pruning and cloud.

Other things

  1. Looked into fitting full joint distributions of hypotheses. This was easy enough to implement, but it's going to take a lot of time to assess how to score them. I've tried a few different methods and nothing consistent is coming out. (Between deciding on bandwidths, and picking points to evaluate the KDEs at, there's a lot of decisions to make.) So I suggest this be tabled.

  2. I found a way of choosing a bin size for the marginal histograms. They follow the mean error on each day exactly (in terms of ordering of hyps). See plots/allNew3/comphisterr for plots. Because of this consistency it's probably again okay to stay with a mean and variance assessment, and we can mention that we get consistent results when measuring marginal histograms.

  3. Distinguishing between sensory context/goals and actual decoder activity: Comparing habitual/pruning hypotheses being fit on thetaGrps vs thetaActualGrps shows that fitting on thetaGrps yields lower error (see issue #183). So it's sensory goals that's better.

Summary

My laptop died and so we had the meeting without me being able to show any plots. Our discussion got a bit messy, but I realized that I kinda changed a lot at once, and somewhere along the line the results seemed to change: the cloud hypothesis is one of the best now!

Here's what changed:

  • no IME --> IME
  • to find error of hypotheses, splitting by cursor-target angle --> splitting by actual cursor velocity (i.e., behavior)
  • to find error of hypotheses, use 8 bins of the above splits --> use 16 bins
  • cloud hypothesis now chooses the closest point in terms of row space activity instead of sampling from nearby points in terms of row space activity

The next-to-last point didn't change any results. But for the others, things do change and it's unclear how all of these combine.

For example, I know that without IME, the second point makes a difference, but I'm not sure about with IME. And without the new cloud, IME vs. no IME didn't make a difference, but the new cloud does change with and without IME! So I just need to cover all of these cases.

Also Byron mentions how maybe we can fit joint distributions in 3d null space, either by taking PCA, or just by choosing randomly 3 dims out of the 8.