Jeff created a PR in sharrow that addresses accessibility calcs
Xarray data format for the skims attaches the origin/destination coordinate locations to various arrays. When running accessibility module, it is pulling out a column of values which has all the data you want, along with an extra copy of the origin/destination zones. This is a massive amount of pointless extra data. PR to sharrow stops the pulling of the coordinates and just adds the raw data that you want.
Inviting reviewer group (RSG and WSP) to test out the changes and confirm it solves all problems and issues are resolved. Code resolution integrated in code that replaces ORCA.
Jeff added an option for explicit chunking, only in the aggregate accessibility component, so far. If you give the explicit chunk setting for the yaml file for that component, then it won’t go through the training/configuration fiddling. If you set it to a reasonable number, then you’ll get a reasonable RAM. In the PSRC model, where you only need chunking in two components, then you can give it a number that works with the machine you are working with, then you’re done. If it fails with a memory error, then it’s obviously the chunking, so you change the number to a lower one and rerun. If you want to do a bunch of fiddling and find the optimal number manually, you can.
Why not set the number of chunks instead of MAZ/TAZs? If we set to a number of chunks, then when we run a test model that has fewer zones, then the test will be slower because it has very few zones in each chunk. However, this might complicate things when running in feedback with different loops have varying sample sizes, where the number of tours generated will vary.
Jeff didn't want to spend a lot of resources to improve this because hopefully the roadmap effort will lead to eliminating the need for chunking; trying to find the simplest solution for now.
Does sharrow work with chunking? Yes, it does. The constraint with sharrow, not specific to chunking but anything where you run the sharrow compiled a lot of times, when the numba compiling happens, it needs to be done in single processing, one time. If there’s multiple processors o, the cache gets corrupted and the program crashes. To avoid this, a pass through of the code in single process without chunking but can be done with a very small sample - this is included in the sharrow documentation.