4 Querying model results - jayfuhrman/gcam-hpc-starterpack GitHub Wiki

Running preset queries

When you run a scenario permutation, the run_model.sh file should call those .xml queries in GCAM-core/output/queries automatically after the model run is finished. Basically, it calls java to read a pointer .xml file (in each exe_* folder from your scenario permutation in scratch) which will in turn point to a corresponding query file (a .xml in GCAM-core/output/queries) that actually runs the query. The .xml query file reads the output data from the model run in exe_0/db/database_basexdb/ (in scratch) and outputs .csv files that are easy to read for each query in each exe_* directory in scratch.

Here is an example for CO2 concentrations globally:

If you want to process model results after a model run and avoid having to re-run the model, you can go to postproc_scratch.sh script to indicate which query you want to run and run the script from the terminal: ./postproc_scratch.sh.

Automating the creation of new queries with python

The queries and pointer files included in this package should be more than sufficient to begin using GCAM and understanding its output data. Therefore, this section should be returned to once you feel comfortable enough working with the model to understand what additional data you might need for your work

In the GCAM-install/java_queries/output/queries directory, a self-documenting python script batch_query_generator.py is provided to automate the creation of new, individual queries. There is also a batch_query_generator_us_state.py file that creates state-level queries for GCAM-USA.

Each of these scripts go into a a file called Main_queries.xml, which contains ALL queries available in the standard release of GCAM. The script will extract the xml element of the individual query you want, puts it in its own file which is written to output/queries, and also creates a second file that points to the new query file. This second file is written to your exe directory.

Finally, it outputs a string which can then be copied and pasted into your run_model.sh file and/or postproc_scratch.sh file, which actually calls java to access the pointer file, which then directs to the query file. (SUPER CONVOLUTED, I KNOW!) It is intended that this will avoid tedium as well as copy/paste errors (specifically with respect to filepaths), and java errors that may arise from them, which may cause issues for new gcam users.

NOTE: These python scripts need to be placed in GCAM-core/output/queries in order to find the filepaths specified in the script. The makescript should do this automatically but it can also be done manually

You can open this script using spyder or another python IDE. Included here is a short shell script to start spyder (spyder.sh)

This script takes as arguments your gcam working directory (GCAM-core), as well as a list of new queries you want to run (query_strs). To create a new query, simply set these to the values you want (for query_str this should be the exact string from the model interface)

Then, run the file. It will generate the pointer file for the query, as well as the query itself, and output a text string which you can copy and paste into the run_model.sh file (for future runs), as well as your postproc_scratch.sh file (if you want to run the query on an already-completed run without rerunning the whole model).