Multiple Zone Systems Tests - ActivitySim/activitysim GitHub Wiki
ActivitySim Multiple Zone Branch Performance Tests
- Objectives:
SANDAG staff (@wusun2) tested the performance of ActivitySim Multiple Zone branch. The objectives of this effort are:
- Evaluate the performance of implemented methods in ActivitySim Multiple Zone Systems by mimicking SANDAG model settings;
- Provide a peek at the overall ActivitySim performance, especially the performance of methods shared by both the Multiple Zone and master ActivitySim branches;
- Identify performance bottlenecks; and
- Summarize findings.
The focus of this effort is on testing performance of critical 'building blocks' that are called numerous times in utility computations, including retrieving zonal TAZ, TAP, and MAZ values, TAZ, TAP, and MAZ skim values, and the transit virtual path building (TVPB) procedure used to find best transit path (BTP).
- Test Environment:
Intel® Xeon® Processor E5-2650 v3 @ 2.30GHz; 256 RAM; 20 cores; Windows Server 2012 R2 Std, 64-bit
- Limitations:
Although the aim of this effort is to test scenarios designed to mimic SANDAG 2012 model settings, the test scenarios don't fully reflect model complexity for the following reasons:
- Many UEC components such as logsums and zonal, household, and personal attributes are not included in the tests.
- Utility expression evaluation tests are not included, expect for TVPB test in finding BTP.
- Although the TVPB test is a comprehensive case somewhat like the implementation in CT-RAMP, the tested TVPB expression is simplified with no probability computations and no Monte Carlo choices.
The performance of a real model scenario could be significantly different from those of the tested scenarios due to the above limitations. Regardless, these tests server as an initial peek at the performance of ActivitySim.
- Approaches:
This section describes the process of creating test scenarios, starting with a breakdown analysis of SANDAG CT-RAMP model components (Table 1). Only San Diego resident models, the most time-consuming part, are included in this analysis. Unlike special market models, the resident models are more generic with similar structures shared by models of most other partner agencies.
For each model component in Table 1, we estimated the computational burden of retrieving skim data by counting the frequency of calling network LOS methods implemented in ActivitySim Multiple Zone Systems branch. The utility expression calculators (UECs) in CT-RAMP provide a vehicle for estimating computational burden, which is affected by the numbers of decision making units (DMU), skims used in each UEC, and alternatives. Each component is then assigned with a low, medium, or high burden levels based on the counts of LOS network method calls. At the end, mandatory tour mode, non-mandatory tour mode, trip mode, and stop location choices are assigned with 'high' burden status; work, school, and non-mandatory tour location choices are assigned with 'medium' burden status; all other components are assigned with 'low' burden status. Based on assigned burden status of each component, we constructed test scenarios following these rules:
- Test work/school location choices; skip non-mandatory tour location choice because its performance is similar to that of work/school location choices.
- Test trip mode choice; skip tour mode choice as it has significantly smaller burden compared with trip mode choice.
- Test stop location choice
- Skip all 'low' burden components
Table 1: SANDAG CT-RAMP Resident Models
| Model Components | DMU | #Units | Skims | Alts | Method | Calls/Alt | Times called | Burden | Test | Note |
|---|---|---|---|---|---|---|---|---|---|---|
| Work location | #workers | 1,313,256 | OP_GP_DIST | 30 | taz_skims() | 1 | 39,397,680 | Med. | Yes | |
| School location | #students | 965,880 | OP_GP_DIST | 30 | taz_skims() | 1 | 28,976,400 | Med. | Yes | |
| Auto ownership | #households | Low | No | Fast | ||||||
| Transponder | #households | Low | No | Fast | ||||||
| Free parking | #workers in CBD | Low | No | Fast | ||||||
| CDAP | #households | Low | No | No Skims | ||||||
| Mandatory tour frequency | #persons with M tours | 2,279,136 | Distances to work/school | * | taz_skims() | 1 | 2,279,136 | Low | No | |
| Mandatory tour mode | #Mandatory tours | 1,913,084 | 100 auto skims | 26 | taz_skims(); TVPB | 4 | High | No | ||
| Mandatory TOD | #Mandatory tours | 1,913,084 | OP_GP_DIST | * | taz_skims() | 1 | 1,913,084 | Low | No | |
| Join Tour frequency | #persons with J tours | Low | No | small size | ||||||
| Join Tour location | #joint tours | Low | No | small size | ||||||
| Join Tour Mode | #joint tours | Low | No | small size | ||||||
| Join Tour TOD | #joint tours | Low | No | small size | ||||||
| Non-Mandatory tour frequency | #persons with N tours | Low | No | No Skims | ||||||
| Non-Mandatory tour location | #Non-Mandatory tours | 1,932,217 | OP_GP_DIST | 30 | taz_skims() | 1 | 57,966,510 | Medium | No | |
| Non-Mandatory tour mode | #Non-Mandatory tours | 1,932,217 | 100 auto skims | 26 | taz_skims(); TVPB | 4 | High | No | ||
| Non-Mandatory TOD | #Non-Mandatory tours | 1,932,217 | OP_GP_DIST | * | taz_skims() | 1 | 1,932,217 | Low | No | |
| AtWork tour frequency | #persons with AtWork tours | Low | No | small size | ||||||
| AtWork tour location | #At-Work tours | Low | No | small size | ||||||
| AtWork tour mode | #At-Work tours | Low | No | small size | ||||||
| AtWork TOD | #At-Work tours | Low | No | small size | ||||||
| Stop frequency | #tours | 3,966,590 | OP_GP_DIST | * | taz_skims() | 1 | 3,966,590 | Low | No | |
| Stop location | #trips | 10,146,986 | distances: os;sd; od; tos; sto | 30 | get_maz_pairs() | 5 | 1,522,047,900 | High | Yes | |
| trip mode choice | #trips | 10,146,986 | 100 auto skims | 26 | taz_skims(); TVPB | 4 | See Table 5 | High | Yes |
- Test Scenarios & Test Sizes:
The relevant method in mandatory tour mode choice is taz_skims(). The test size is set to 68,374,080 (Table 2), estimated from the numbers of workers and students, skims used in UEC, and choice alternatives (30 location alternative samples).
Table 2: Mandatory Tour Choice Test
| Variables affect test size | |
|---|---|
| # workers | 1,313,256 |
| #students | 965,880 |
| # skim calls in UEC per unit per alternative | 1 |
| skim | off peak distance |
| method | taz_skims() |
| #alternatives | 30 |
| Test Size | 68,374,080 |
The relevant method in stop location choice is get_maz_pairs(). The test size is set to 1,522,047,900 (Table 3), estimated from the numbers of trips, skims used in UEC, and choice alternatives (30 location alternative samples).
Table 3: Stop Location Choice Test
| Variables affect test size | |
|---|---|
| # trips | 10,146,986 |
| # skim calls in UEC per trip per alternative | 5 |
| skim | Distances: O to S, S to D, O to D, tour O to S, and S to tour D |
| method | get_maz_pairs() |
| #alternatives | 30 |
| Test Size | 1,522,047,900 |
The test size estimation for trip mode choice model is more nuanced. Trip mode choice UEC utilizes auto, walk, bike, and transit skims (Table 4). There are 100 auto skims alone used in SANDAG CT-RAMP. However, the utility computation of each mode utilizes only a few relevant skims specific to that mode. For example, the only skim needed for computing walk mode utility is walk time. For transit modes, instead of transit skims, the utility relies on the BTP found via the TVPB procedure.
Table 4: Trip Mode Choice Skims
| Method | Skims | ||
|---|---|---|---|
| #trips | 10,146,986 | ||
| # auto skim calls in UEC per trip per alternative | 4 | taz_skims() | time and distance by inbound/outbound time period |
| # walk skim calls | 1 | get_maz_pairs() | walk time |
| # bike skim calls | 1 | get_maz_pairs() | bike time |
| # TVPB Calls | 1 | TVPB | best transit path |
| # auto alternatives | 14 |
The burden of computing mode choice utility is mitigated because many trips only have access to a subset of the 26 mode alternatives due to the maximum allowed walk, bike, walk to transit, and drive to transit time thresholds. However, this also makes it difficult to estimate how many modes are available to a trip because mode availability depends on trip origin and destination. In this effort, we constructed 4 hypothetical trip mode choice scenarios (Table 5) that represent various levels of mode availability to trips:
- 10% trips have access to transit; 25% trips have access to walk and bike; 100% trips have access to drive
- 25% trips have access to transit; 50% trips have access to walk and bike; 100% trips have access to drive
- 33% trips have access to transit; 60% trips have access to walk and bike; 100% trips have access to drive
- 50% trips have access to transit; 75% trips have access to walk and bike; 100% trips have access to drive
The frequency of method calls (Table 5) are derived from the total number of trips and mode availability assumptions above. For example, taz_skims(), get_maz_pairs() and TVPB are called 568,231,216 (, 5,073,493 (100% of all 10,146,986 trips have access to drive mode, each trip has 14 auto mode alternatives, and each alternative uses 4 skims), 2,536,747 bike and walk distance calls each), and 1,014,699 times respectively in scenario 1.
Table 5: Trip Mode Choice Test Scenarios
| Method | Trip Mode 1 | Trip Mode 2 | Trip Mode 3 | Trip Mode 4 | |
|---|---|---|---|---|---|
| transit available trips | 10% | 25% | 33% | 50% | |
| walk available trips | 25% | 50% | 60% | 75% | |
| bike available trips | 25% | 50% | 60% | 75% | |
| drivable trips | 100% | 100% | 100% | 100% | |
| # auto skims calls | taz_skims() | 568,231,216 | 568,231,216 | 568,231,216 | 568,231,216 |
| # walk skim calls | get_maz_pairs() | 2,536,747 | 5,073,493 | 6,088,192 | 7,610,240 |
| # bike skim calls | get_maz_pairs() | 2,536,747 | 5,073,493 | 6,088,192 | 7,610,240 |
| # TVPB Calls | TVPB | 1,014,699 | 2,536,747 | 3,348,505 | 5,073,493 |
Based on the analysis in tables 2, 3 and 5, the test sizes of relevant methods in mandatory location, stop location, and trip mode choice models are summarized in Table 6.
Table 6: Test Scenarios & Test Sizes
| Model Component | Relevant Method | Test Size |
|---|---|---|
| Mandatory tour mode choice | taz_skims() | 68,374,080 |
| Stop location choice | get_maz_pairs() | 1,522,047,900 |
| Trip mode choice 1 | taz_skims() | 568,231,216 |
| get_maz_pairs() | 5,073,493 | |
| best transit path | 1,014,699 | |
| Trip mode choice 2 | taz_skims() | 568,231,216 |
| get_maz_pairs() | 10,146,986 | |
| best transit path | 2,536,747 | |
| Trip mode choice 3 | taz_skims() | 568,231,216 |
| get_maz_pairs() | 12,176,383 | |
| best transit path | 3,348,505 | |
| Trip mode choice 4 | taz_skims() | 568,231,216 |
| get_maz_pairs() | 15,220,479 | |
| best transit path | 5,073,493 |
- Test Results:
Table 7 is the performance summary of data loading. In this test, population, land use, and walk and bike impedance data are loaded. Although SANDAG model uses 80 skims (70 auto and 10 transit skims), only 6 skims are loaded in this test, including AM SOV, PM SOV, AM local bus, AM premium bus, PM local bus and PM premium skims. Data loading performs well, taking less than 30 seconds.
Table 7: Data Loading Performance
| Test Item | Runtime (sec) |
|---|---|
| Load TAZ skims | 16.3 |
| Load TAP skims | 10.5 |
| Load network LOS | 2.9 |
| Total | 29.7 |
Table 8 is a performance summary of each individual method by increasing test size from 10,000 to 50,000,000.
Table 8: Individual Method Performance
| Runtime | |||||
|---|---|---|---|---|---|
| Test size | 10,000 | 100,000 | 1,000,000 | 10,000,000 | 50,000,000 |
| get_taz() | 0.008 | 0.048 | 0.722 | 40.92 | 1030.5 |
| get_tap() | 0.007 | 0.054 | 0.808 | 42.29 | 1030.3 |
| get_maz() | 0.023 | 0.183 | 1.695 | 53.53 | 1089.8 |
| taz_skims() | 0.004 | 0.034 | 0.326 | 3.36 | 18.8 |
| tap_skims() | 0.014 | 0.135 | 1.738 | 94.17 | 2188.7 |
| get_maz_pairs() | 2.867 | 1.213 | 5.367 | 78.57 | 1142.3 |
| get_maz_tap_pairs() | 0.084 | 0.244 | 1.928 | 50.73 | 1127.4 |
| get_taps_mazs() | 0.118 | 0.341 | 2.749 | 28.84 | 159 |
Table 9 is a performance summary of each individual method per unit (10000 as one unit) derived from Table 8.
Table 9: Individual Method Performance per Unit (10000)
| Runtime/10000 | |||||
|---|---|---|---|---|---|
| Test size | 10,000 | 100,000 | 1,000,000 | 10,000,000 | 50,000,000 |
| get_taz() | 0.008 | 0.005 | 0.007 | 0.041 | 0.206 |
| get_tap() | 0.007 | 0.005 | 0.008 | 0.042 | 0.206 |
| get_maz() | 0.023 | 0.018 | 0.017 | 0.054 | 0.218 |
| taz_skims() | 0.004 | 0.003 | 0.003 | 0.003 | 0.004 |
| tap_skims() | 0.014 | 0.014 | 0.017 | 0.094 | 0.438 |
| get_maz_pairs() | 2.867 | 0.121 | 0.054 | 0.079 | 0.228 |
| get_maz_tap_pairs() | 0.084 | 0.024 | 0.019 | 0.051 | 0.225 |
| get_taps_mazs() | 0.118 | 0.034 | 0.027 | 0.029 | 0.032 |
Table 10 is a performance summary of relevant methods by model component.
Table 10: Method Performance by Model Component
| Model Component | Relevant Method | Test Size | Runtime (secs) | Runtime(hrs) |
|---|---|---|---|---|
| Mandatory tour mode choice | taz_skims() | 68,374,080 | 24.01 | 0.01 |
| Stop location choice | get_maz_pairs() | 1,522,047,900 | Killed at 68th hr | |
| Trip mode choice scenario 1 | taz_skims() | 568,231,216 | 209.3 | 0.06 |
| get_maz_pairs() | 5,073,493 | 35.6 | 0.01 | |
| best transit path | 1,014,699 | 23391.2 | 6.50 | |
| Total | 23636.1 | 6.57 | ||
| Trip mode choice scenario 2 | taz_skims() | 568,231,216 | 209.3 | 0.06 |
| get_maz_pairs() | 10,146,986 | 83.4 | 0.02 | |
| best transit path | 2,536,747 | killed at 12th hr | ||
| Total | ||||
| Trip mode choice scenario 3 | taz_skims() | 568,231,216 | 209.3 | 0.06 |
| get_maz_pairs() | 12,176,383 | 114 | 0.03 | |
| best transit path | 3,348,505 | NA | ||
| Total | ||||
| Trip mode choice scenario 4 | taz_skims() | 568,231,216 | 209.3 | 0.06 |
| get_maz_pairs() | 15,220,479 | 161.4 | 0.04 | |
| best transit path | 5,073,493 | NA | ||
| Total |
- Findings:
- Data loading performs well. Loading household, population, land use, walk and bike impedances, and 6 auto/transit skims took less than 30 seconds.
- In mandatory tour location choice, taz_skims() method performs well, taking 24 seconds.
- In stop location choice, the get_maz_pairs() method is a bottleneck. We were not able to complete get_maz_paris() test with test size 1,522,047,900 because the runtime was too long.
- In trip mode choice, the TVPB procedure is a bottleneck. The TVPB calls took 6.5 hours in the most conservative trip mode scenario. We were not able to complete the TVPB tests for the other three trip mode scenarios because the runtime was too long.
- As test size increases, the per 10000 unit runtime of get_taz(), get_tap(), get_maz(), tap_skims(), and get_maz_tap_pairs() methods increases significantly.
- As test size increases, the per 10000 unit runtime of get_maz_tap_pairs() method fluctuates with a unexpected large runtime when the test size is 10,000.
- As test size increases, the per 10000 unit runtime of taz_skims() and get_taps_mazs() method is stable; Method taz_skims() consistently performs well even when test size is large.
- The less than ideal performance of tap_skims(), in comparison with taz_skims(), is particularly interesting. If tap_skims() can be implemented in a way similar to taz_skims(), the overall ActivitySim performance could be improved.