SubstImprovement2017 - ajf58/scons GitHub Wiki
Thoughts on possible improvements for Subst().
- There are few unique strings stored among all Environment()'s created or cloned
- Most cloned Environments are mostly the same as all (or many others)
- Subst() is called several orders of magnitude more than there are command lines and/or unique strings
- The following are potentially the only unique items for most command lines (and are thus not cacheable) list from SCons.Environment.reserved_construction_var_names
- CHANGED_SOURCES
- CHANGED_TARGETS
- SOURCE
- SOURCES
- TARGET
- TARGETS
- UNCHANGED_SOURCES
- UNCHANGED_TARGETS
- callable values in Environment's dict.
- The subst logic is called with a dictionary of local and global values and the string (or list) to be evaluated
- Most of the time the global dictionary passed is the values from the Environment() being used by the task to build the target (or to evaluate if the target is out of date which requires the command line string to be evaluated). The local dictionary passed may be the TARGET and SOURCE information.
- The subst logic often involves recursive calls to itself or it's peers (string and list versions of subst)
- Subst current is composed of two steps
- tokenize the string (currently via regex)
- recursively evaluate the string
- Subst can be called in two basic modes
- create a command line
- create a string for comparing signature (this drops items between
$( and $ ) )
- If the evaluated string is a callable, it is called as follows
#!python
callable(target,source,env,for_signature)
- Tokenize Environment() variables when they are set.
- Scan tokens to identify directly accessed variables. Add these to set which will be checked to invalidate cache when other variables are set. (Including list above)
- Consider altering strings generated for signatures to exclude SOURCE and TARGET unless they are modified (.abspath,etc ). This would allow using pre-expanded variables (for example CXXCOM would have everything but TARGET and SOURCE pre expanded, so for signatures this should be a LOT faster)
- On env.Clone() make shallow copy of environment variables, and then use copy-on-write if the variable changed