Postprocessors - mf4dl1/jaksafe GitHub Wiki
This document explains the purpose of postprocessors and lists the different available postprocessor and the requirements each has to be used effectively.
Note
This document is still a work in progress.
A postprocessor is a function that takes the results from the impact function and calculates derivative indicators, for example if you have an affected population total, the Gender postprocessor will calculate gender specific indicators such as additional nutritional requirements for pregnant women. InaSAFE works by creating an intermediate vector layer that contains result of the aggregation done for your impact analysis. For example if you are doing a flood impact on OSM buildings and using an aggregation layer with a few polygons, the intermediary layer will contain the the number of buildings flooded for each aggregation polygon. It is recommended, for understanding purpose, to enable in InaSAFE options "Show intermediate layers generated by the post processing" and inspect the attributes table. If no aggregation layer is given, the area of interest will be used to create a layer with a single polygon.
Adding a new postprocessor is as simple as adding a new class called XxxxxxPostprocessor that inherits AbstractPostprocessor with 2 mandatory methods (process, description), 2 optional ones and as many indicators as you need.
the minimal class could look like this:
class MySuperPostprocessor(AbstractPostprocessor):
def __init__(self):
AbstractPostprocessor.__init__(self)
def description(self):
return 'mydescription'
def setup(self, params):
AbstractPostprocessor.setup(self, None)
def process(self):
AbstractPostprocessor.process(self)
self._calculate_my_indicator()
def clear(self):
AbstractPostprocessor.clear(self)
def _calculate_my_indicator(self):
x = 5
A = 0.5
myResult = 10 * x / A
self._append_result('My Indicator', myResult)
After that you need to import the new class into postprocessor_factory and update AVAILABLE_POSTPROCESSORS to include the postprocessor prefix (e.g. MySuper if the class is called MySuperPostprocessor and its human readable name)
from mysuper_postprocessor import MySuperPostprocessor
AVAILABLE_POSTPTOCESSORS = {'Gender': 'Gender',
'Age': 'Age',
'Aggregation': 'Aggregation',
'BuildingType': 'Building type',
'AggregationCategorical':
'Aggregation categorical',
'MinimumNeeds': 'Minimum needs',
'MySuper': 'My Super Postprocessor'
}
from mysuper_postprocessor import MySuperPostprocessor
AVAILABLE_POSTPTOCESSORS = {'Gender': 'Gender',
'Age': 'Age',
'Aggregation': 'Aggregation',
'BuildingType': 'Building type',
'AggregationCategorical':
'Aggregation categorical',
'MinimumNeeds': 'Minimum needs',
'MySuper': 'My Super Postprocessor'
}
As last step you have to update or add the parameters variable to the impact functions that you want to use in the new postprocessor. This will need to include a dictionary of the available postprocessors as shown below.
parameters = {
'thresholds': [0.3, 0.5, 1.0],
'postprocessors':
{'Gender': {'on': True},
'Age': {'on': True,
'params': {
'youth_ratio': defaults['YOUTH_RATIO'],
'adult_ratio': defaults['ADULT_RATIO'],
'elder_ratio': defaults['ELDER_RATIO']
}
},
'MySuper': {'on': True}
}
}
or as a minimum:
parameters = {'postprocessors':{'MySuper': {'on': True}}}
If your post processor runs successfully and produces a result, this result will be appended to the analysis result. You can use one of the impact functions (e.g. : flood_OSM_building_impact) to add your test processor and test it. You should see a section containing the result of your super postprocessor:
For implementation examples see AgePostprocessor, GenderPostprocessor and BuildingTypePostprocessor which use mandatory and optional parameters.
- statistics_type = 'sum'
- statistics_type = 'class_count'
Impact layers produced by InaSAFE can either be raster or vector type (depending on the exposure data used as input). When doing an aggregation, each feature in the intermediate layer will contain the result of the aggregation. The way the aggregation is calculated depend on the type of the impact layer and on the impact function that was used to produce the impact layer.
For vector layers, two type of aggregations are possible:
- Sum aggregation: this will sum up into one field the number of exposure data that is part of the aggregation polygon. Impact functions are by default set to use this type of aggregation.
- Class count aggregation: It is also possible to define the intermediate aggregation layer to contain the number of items for a series of valued defined in the impact function. Please refer to the earth_building_impact function and see the use of statistics_type = 'class_count' and statistics_classes.
For raster layers, the aggregation (sum, count, mean) is done using QGIS's zonal statistics functionality (please refer to zonal_stats.py and https://github.com/qgis/Quantum-GIS/blob/master/src/analysis/vector/qgszonalstatistics.cpp for details.
Depending of the intermediate aggregate layer produced, the post processor would need use the resulting aggregation values and attributes to do it's calculation.
It is interesting to review some of the code in this post processor that is used to produce a report of affected buildings by type. The setup method is called for each aggregation polygon. It is called with all the necessary parameters that are needed by the process method to classify buildings by type. This aggregator is expecting to work on aggregation done as a sum on vector impact layer.
def setup(self, params):
"""Initialises parameters.
"""
AbstractPostprocessor.setup(self, None)
if (self.impact_total is not None or
self.impact_attrs is not None or
self.target_field is not None or
self.valid_type_fields is not None or
self.type_fields is not None):
self._raise_error('clear needs to be called before setup')
self.impact_total = params['impact_total']
self.impact_attrs = params['impact_attrs']
self.target_field = params['target_field']
self.valid_type_fields = params['key_attribute']
- impact_total: the total number of buildings that are contained in the polygon
- impact_attrs: all attributes for all the features contained in the polygon
- target_field: attribute name used to detect the status of the building. In the case of the flood impact (flood_OSM_building_impact), the attribute is INUNDATED. This attribute was set and written as part of the keywords by the impact function.
- key_attribute: attribute name that is used to get the type of the building. This can either be set by the impact function or the default name 'type' will be used.
Note
key_attribute is for now only available for the BuildingType processor. To adjust/review this, please refer to postprocessor_manager class.
This aggregator is expected to work on aggregation done on a raster impact layer. Looking at the setup method, it is important to understand that the parameter impact_total will contain the aggregated value (normally the number of people of a particular aggregation area)
def setup(self, params): self.impact_total = params['impact_total'] ... #either all 3 ratio are custom set or we use defaults self.youth_ratio = params['youth_ratio'] self.adult_ratio = params['adult_ratio'] self.elder_ratio = params['elder_ratio']
AggregationCategoricalPostprocessor is used with impact functions that are setup to do class count aggregation (see section :ref:`types_of_aggregation`). An example of such impact function is the EarthquakeBuildingImpactFunction where four class types (levels of hazard) are defined. Looking into the setup method, it is important to understand that the impact_classes parameter contains these classes.
def setup(self, params): ... self.impact_classes = params['impact_classes'] ...
InaSAFE provides a post processor (MinimumNeedsPostprocessor) that will use a series of parameters to quickly calculate the needs of displaced people (e.g. in terms of drinking water, food, ...). Please refer to :ref:`minimum_needs` on notes related to this functionality. Couple of interesting points to mention regarding the use of this post processor:
- Impact functions need to define the minimum needs as part of their parameters.
For example:
parameters = OrderedDict([
('thresholds [m]', [1.0]),
('postprocessors', OrderedDict([
('Gender', {'on': True}),
('Age', {
'on': True,
'params': OrderedDict([
('youth_ratio', defaults['YOUTH_RATIO']),
('adult_ratio', defaults['ADULT_RATIO']),
('elder_ratio', defaults['ELDER_RATIO'])])}),
('MinimumNeeds', {'on': True}),
]))
('minimum needs', default_minimum_needs())
])
- The parameters defined as part of minimum needs are not yet configurable by the user. If there is a need to make a change, you can either define them inside the impact functions or modify the default needs defined in core.py.
Impact layers produced by InaSAFE can either be raster or vector type (depending on the exposure data used as input). When doing an aggregation, each feature in the intermediate layer will contain the result of the aggregation. The way the aggregation is calculated depend on the type of the impact layer and on the impact function that was used to produce the impact layer.
For vector layers, two type of aggregations are possible:
- Sum aggregation: this will sum up into one field the number of exposure data that is part of the aggregation polygon. Impact functions are by default set to use this type of aggregation.
- Class count aggregation: It is also possible to define the intermediate aggregation layer to contain the number of items for a series of valued defined in the impact function. Please refer to the earth_building_impact function and see the use of statistics_type = 'class_count' and statistics_classes.
For raster layers, the aggregation (sum, count, mean) is done using QGIS's zonal statistics functionality (please refer to zonal_stats.py and https://github.com/qgis/Quantum-GIS/blob/master/src/analysis/vector/qgszonalstatistics.cpp for details.
Depending of the intermediate aggregate layer produced, the post processor would need use the resulting aggregation values and attributes to do It's calculation.
It is interesting to review some of the code in this post processor that is used to produce a report of affected buildings by type. The setup method is called for each aggregation polygon. It is called with all the necessary parameters that are needed by the process method to classify buildings by type. This aggregator is expecting to work on aggregation done as a sum on vector impact layer.
def setup(self, params):
"""Initialises parameters.
"""
AbstractPostprocessor.setup(self, None)
if (self.impact_total is not None or
self.impact_attrs is not None or
self.target_field is not None or
self.valid_type_fields is not None or
self.type_fields is not None):
self._raise_error('clear needs to be called before setup')
self.impact_total = params['impact_total']
self.impact_attrs = params['impact_attrs']
self.target_field = params['target_field']
self.valid_type_fields = params['key_attribute']
-
impact_total: the total number of buildings that are contained in the polygon
-
impact_attrs: all attributes for all the features contained in the polygon
-
target_field: attribute name used to detect the status of the building. In the case of the flood impact (flood_OSM_building_impact), the attribute is INUNDATED. This attribute was set and written as part of the keywords by the impact function.
-
key_attribute: attribute name that is used to get the type of the building. This can either be set by the impact function or the default name 'type' will be used.
Note
key_attribute is for now only available for the BuildingType processor. To adjust/review this, please refer to postprocessor_manager class.
This aggregator is expected to work on aggregation done on a raster impact layer. Looking at the setup method, it is important to understand that the parameter impact_total will contain the aggregated value (normally the number of people of a particular aggregation area). It also uses a series of user configurable parameters that are used for ratio calculations. Other post processors such as GenderPostprocessor and MinimumNeedsPostprocessor are based on the same logic with a total impact number and customizable parameters.
def setup(self, params): ... self.impact_total = params['impact_total'] ... #either all 3 ratio are custom set or we use defaults self.youth_ratio = params['youth_ratio'] self.adult_ratio = params['adult_ratio'] self.elder_ratio = params['elder_ratio']
AggregationCategoricalPostprocessor is used with impact functions that are setup to do class count aggregation (see section :ref:`types_of_aggreagation`). An example of such impact function is the EarthquakeBuildingImpactFunction where four class types (levels of hazard) are defined. Looking into the setup method, it is important to understand that the impact_classes parameter contains these classes.
def setup(self, params): ... self.impact_classes = params['impact_classes'] ...
InaSAFE provides a post processor (MinimumNeedsPostprocessor) that will use a series of parameters to quickly calculate the needs of displaced people (e.g. in terms of drinking water, food, ...). Please refer to :ref:`minimum_needs` on notes related to this functionality. Couple of interesting points to mention regarding the use of this post processor:
- impact functions need to define the minimum needs as part of their parameters. For example:
....
parameters = OrderedDict([
('thresholds [m]', [1.0]),
('postprocessors', OrderedDict([
('Gender', {'on': True}),
('Age', {
'on': True,
'params': OrderedDict([
('youth_ratio', defaults['YOUTH_RATIO']),
('adult_ratio', defaults['ADULT_RATIO']),
('elder_ratio', defaults['ELDER_RATIO'])])}),
('MinimumNeeds', {'on': True}),
]))
('minimum needs', default_minimum_needs())
])
- the parameters defined as part of minimum needs are not yet configurable by the user. If there is a need to make a change, you can either define them inside the impact functions or modify the default needs defined in core.py.
Dock.postproc Output will hold the result datastructure (shown below) of all the postprocessors. The structure is then parsed by Dock._postProcessingOutput() and stored in the impact layer's keywords. If a postprocessor generates no output (for example due to calculation errors), then it will just be skipped from the report.
{'Gender': [
(QString(u'JAKARTA BARAT'), OrderedDict([(u'Total', {'value': 278349, 'metadata': {}}),
(u'Females count', {'value': 144741, 'metadata': {}}),
(u'Females weekly hygiene packs', {'value': 114881, 'metadata': {'description': 'Females hygiene packs for weekly use'}})])),
(QString(u'JAKARTA UTARA'), OrderedDict([(u'Total', {'value': 344655, 'metadata': {}}),
(u'Females count', {'value': 179221, 'metadata': {}}),
(u'Females weekly hygiene packs', {'value': 142247, 'metadata': {'description': 'Females hygiene packs for weekly use'}})]))],
'Age': [
(QString(u'JAKARTA BARAT'), OrderedDict([(u'Total', {'value': 278349, 'metadata': {}}),
(u'Youth count', {'value': 73206, 'metadata': {}}),
(u'Adult count', {'value': 183432, 'metadata': {}}),
(u'Elderly count', {'value': 21990, 'metadata': {}})])),
(QString(u'JAKARTA UTARA'), OrderedDict([(u'Total', {'value': 344655, 'metadata': {}}),
(u'Youth count', {'value': 90644, 'metadata': {}}),
(u'Adult count', {'value': 227128, 'metadata': {}}),
(u'Elderly count', {'value': 27228, 'metadata': {}})]))
]
}