Speed of compiling could be do to CPU configuration, depends on how many optimizations the compiler is attempting by CPU. Running on servers with Xeon chips may take longer as well. Heavy duty servers gain their speed advantage by having a lot of processing cores, but they may not be individually very fast. Number of cores doesn’t matter as much as the speed of the individual cores.
We should test by more people on different machines.
Jeff is working on work flow to automating process to compile, chunk train, production mode, and compare all three.
Running the compiler in multiprocessing is generally not a good idea.
Are there model components that sharrow is not implemented?
Scheduling ones – the logsums have been sharrow-ed but scheduling model has time windows code that hasn’t been moved into numba so sharrow can’t understand the time windows code in the specs. It is the last task of CS’ phase 7 scope to fix that, just hasn’t happened yet.
Composition model might also have the time windows code.
Components without choice models won’t be sharrowed.
Looking at the config file, the sharrow keyword had different options (test versus require, for example). Here are the settings:
False: it is not invoked
True: it will be invoked where it is compatible. If any error is encountered when sharrow is in process, it will fall back to legacy model. Good for compatibility, not good for performance.
Test: runs sharrow but defaults to legacy code if it doesn’t work somewhere. If you have test message, there may be warnings that pop up. If there are deviations between outputs of legacy and sharrow versions. It will stop if there are significant differences.
Require: if sharrow is attempted for use, and it fails, it will crash the run and report errors (let you know where it isn’t working). As opposed to True, which won’t fail and will default to legacy version.
In the long-term, recommend moving away from test mode because it takes a very long time, running legacy and sharrow versions.
Reduces RAM significantly but does take time (about 20 minutes)
Opportunity to do this better, just takes time and resources
Discussed today (and previously) was Jeff’s method in order to allow him to test things on his laptop, by keeping the RAM low. Helps, but can be done better.
If we don’t need to keep floating point precision (boardings, for example) can there be some preprocessing to make the skims themselves smaller, instead of manipulating if it isn’t needed?
This could be a topic for the technical working group, it has some design implications.