Loading from scratch - DiGi-Organization/DiGi.GIS.PostgreSQL.UI GitHub Wiki
The order of tray-app tasks that turns a BDOT10k export (Geoportal Polska_GML.zip) into a database whose derived tables - building_2d.subdivision_id, the occupancy tables, building_data - carry the rules the code now implements. Every rule lives on the code paths a fresh load runs through; what is not automatic is the order, and one step in it is easy to skip.
This page is the operator entry point for a rebuild. The rules themselves are documented in DiGi.Maintenance/documentation/AI Guidelines/Coding - GIS Administrative Data.md; the issues that shaped them are linked at the end.
Important
A fresh building_2d row has no subdivision_id. The loader never derives it; the only writer of that column is Refresh Building2Ds. Skip that step and nothing is wrong, but the occupancy and building-data tasks skip every building whose column is NULL - the tables come out empty, not incorrect.
| # | Tray app task | Writes | Cost (Poland) |
|---|---|---|---|
| 0 | Create main database, Create storage database, Create AdministrativeAreal2D table, Create Building2D table | schema | seconds |
| 1 |
Upload Areal2Ds from BDOT10k file (administrative areas from the .zip), or Upload AdministrativeAreal2Ds from file
|
administrative_areal_2d - one row per polygon part, type_id from the unit's class, object with the polygon and the raw liczbaMieszkancow (null when the export has none) |
minutes |
| 2 | Refresh AdministrativeAreal2Ds | the parent chain of every row (country_id / voivodeship_id / county_id / municipality_id), rebuilt from code and geometry |
minutes |
| 3 | Upload Areal2Ds from BDOT10k file (buildings), or Upload Building2Ds from file |
building_2d - county_id from the part the building sits on, boxes, object; subdivision_id NULL
|
hours |
| 4 | Refresh Building2Ds - defaults: override off, nested-only off, no county list |
building_2d.subdivision_id = the smallest subdivision containing the building
|
an hour or two |
| 5 | Update occupancy from database - both sides, clear on |
occupancy_data_administrative_areal_2d (the roll-up) and occupancy_data_building_2d (per building) |
minutes to an hour |
| 6 |
Update building data - General, Database, Occupancy, RadialRatios, Statistical
|
building_data |
hours |
| 7 | Refresh OrtoDatas → the download loop; then the imagery-dependent pipelines (CityGML buildings, building models, terrain, year built) | their own tables | days |
| 8 | Update external components area - the counties and the statement timeout are asked for when the task is started | the 35 External Components Area columns of building_data (external wall, roof and floor areas by direction and roof tilt), from the stored building_model_component models |
per county, minutes to an hour |
Step 1 must precede 4: the refresh loads each county's subdivision layer by the county part's box. Step 2 must precede 5 and 6: they find a county's subdivisions through county_id on the subdivision rows, which is what the administrative refresh writes. Step 4 must precede 5 and 6 for the reason in the box above. Nothing after step 4 rewrites the column - re-posting a building keeps the stored value (COALESCE(EXCLUDED.subdivision_id, subdivision_id)). Step 8 needs the models step 7's pipelines store: it reads the stored building models and classifies their components, so it runs after them; a county without stored models is processed, not failed - its buildings keep their current values.
The Subdivision layer nests, nearly everywhere. A village and its named parts, a town and its same-named unit, a city and its districts and their neighbourhoods are all type-4 rows of one municipality; 404 of the 406 county parts nest. Warsaw's layer is 217 rows deep to five levels. Everything below is written for that, so the export's shape does not matter to the rules - only to the figures.
Districts arrive as type 4 too. district_or_delegation and town_in_urban_rural_municipality are AdministrativeDivision objects stored under the subdivision type (Query.AdministrativeArealType). Warsaw has 18 of them, coincident with the 18 part_of_city district polygons and carrying no figure. They are never picked for a building (the pick uses divisions only when no subdivision candidate exists), they are not part of the occupancy layer, and they nest under the city so no roll-up counts them. Expect them; do not "clean them up".
subdivision_id (step 4). Each building goes to the smallest subdivision containing it - the most specific unit - decided in memory per county (SubdivisionIdSolver), with a fallback to the database path for a building no layer places. Before this rule the tie between nested containers went to the lowest identifier, which filed all of Warsaw under 49 rows and 57 551 buildings under the city itself. The finish log line names the rows read, written, the fallbacks and the per-county counts:
Building2DPostgreSQLConverter refresh finished: 155307 records read, 155307 subdivision IDs written, 0 resolved against the database, 0 batches stepped over, ...
Occupancy (step 5). Administrative side: every subdivision keeps its own figure, null stays null; a municipality is the sum of its top-level subdivisions only (Query.ContainerIds), a county the sum of its municipalities. Building side: a building takes its share from the smallest figured subdivision containing it (Query.OccupancySubdivisionId), and that subdivision's figure is spread by floor area over every building inside it. A subdivision with no figure anywhere up its chain writes nothing for its buildings - MissingOccupancySubdivisionCount; one that borrowed a container's figure is BorrowedOccupancySubdivisionCount. Both are in the finish log.
Building data (step 6). Groups by the column and reads the subdivision's name, occupancy and settlement type from it - correct by construction once step 4 has run. The administrative columns (Subdivision Id, Subdivision name, Subdivision occupancy, Settlement type) are written by the General update type; a run with Occupancy alone refreshes Calculated occupancy from the occupancy table and leaves the rest as it was - so a re-run after step 4 needs General in the selection. The three year-built columns are a separate update type (YearBuilt), written only after the year-built pipeline (step 7) has stored its history: Predicted year built and User year built each hold the most frequent value of their source (the prediction, or the user entries), and Calculated year built is the user entry when one exists, otherwise the predicted year.
Coverage reads (gis/ortodatas/estimatedcoveragefactor) never use the column: an area is measured over its own polygon, a municipality included. They work the moment steps 1 and 3 are in.
All reads against the API, none against a .conf (see Coding - Deployed WebAPI.md).
| Check | Read | Expect |
|---|---|---|
| the layer is in | gis/administrativeareal2d/administrativeareal2Dreferencesbyadministrativearealtype?administrativearealtype=4 |
~100 000 references, CountyId set on every one (step 2) |
| the column is filled | gis/building2d/building2Dreferencesbyadministrativeareal2Did?administrativeareal2Did=<municipality id> |
every reference carries a SubdivisionId; for a nested municipality the ids used span the whole layer, and the city/village rows hold only the buildings none of their parts hold |
| by column vs by polygon | the same read for a subdivision id (polygon-based) against the count of that id in the municipality read | equal up to the buildings straddling its boundary (Jelonki: 2 240 by polygon, 2 237 by column) |
| the roll-up | gis/occupancydata/administrativeareal2d/itemsbyreference?reference=<municipality reference> |
the city's own figure for a nested city (m. St. Warszawa 1 622 594, not the 4 605 603 a sum over all rows gives) |
| a building's share |
gis/occupancydata/building2d/itemsbyreference?reference=<building>&countyid=<id> and gis/buildingdata/tablebyreference?reference=<building>&countyid=<id>
|
a row for a building in a figure-less neighbourhood; Subdivision name naming the smallest unit |
building_data complete |
gis/buildingdata/coveragebycountyid?countyid=<id> |
MissingReferenceCount 0, UnassignedSubdivisionCount 0 |
-
Two tray apps look the same. The deployed copy under the software directory and the repository
binshow the same tray icon; check the processPathbefore trusting where a task ran, and read the log of the folder it was started from. - A county code is not a key. 18 counties are several polygon parts, one row each; the tasks take part ids. A code endpoint collapses to the lowest part.
-
Run steps 4 and 5 with the app that carries the current build. The pick and the pool rules are in
DiGi.GIS.PostgreSQL; an older tray app runs the older rules against the same database without complaint. -
orto_datas.subdivision_idis a copy of the building's column taken at download or Refresh OrtoDatas time; it is NULL for everything downloaded before step 4 ran. Nothing reads it for a measurement (subdivisionlinksbycountyidreports the gap); it is not a reason to re-download. - Figures do not reconcile across levels. Districts' figures need not equal the sum of their neighbourhoods' (Mokotów 194 777 against 355 214). The building side is built to stay bounded regardless; do not expect per-building totals to add up to a district.
-
Multi-part counties. After step 3 read
gis/building2d/countypartmismatchesfor the 18 multi-part codes before going on: a building filed under a sibling part reads back missing from every table keyed on(county_id, reference). The history is #68 and the guideline's section 3.
- Rules:
DiGi.Maintenance/documentation/AI Guidelines/Coding - GIS Administrative Data.md - Attribution to the smallest container: DiGi.GIS.PostgreSQL#77; runs and evidence: #78; in-memory refresh: #79; occupancy from the smallest figured container: #80; polygon reads: #75, #76
- Parent chain of the administrative rows: #14, #15
- Point-in-polygon at scale: DiGi.Geometry#5
- Year built pipeline, once the imagery is in: Running a County