we looked into Luigi and Airflow but decided to stick with orca for now even though it isn't widely adopted
Example updated and partially run on the full example as well (used all available RAM 160GB)
todos:
write code docs
update ActivitySim documentation, including documentation on data table dependencies for model steps #168
get running on full example; maybe we should just write the additional fields after each model step?
make the current saving table-states debug statements prettier and put in the log file #160
@Everyone agreed that it would be good to clean-up the reading and processing of input data (i.e. the default namespace) now that pipelining is implemented. #169 This means:
creating new model steps for loading and processing inputs
making the default tables just tables with key columns
moving all the orca computed columns to a mtctm1 folder/namespace for example
We will NOT recode the orca computed columns as expressions files since the perceived benefit of this isn't great and it is a big effort
It needs to happen eventually, especially for someone trying to use the code on their own model
Running the full example is pretty slow and using 3% of the CPU power. Maybe we should experiment with multi-processing in order to gauge the data pipeline performance? But this is no small task...
add random numbers for each model for each chooser to the relevant tables #163 - households get auto own rnum for example. This is lower priority for now.