Version 2023 Release Notes - danmarksmiljoeportal/DkJord GitHub Wiki

Version 2023, also commonly referred to as 23x or ÆØ597, introduced major changes to both the system architecture as well as to the data model and service model towards third party systems. The following release note explains all the main changes of version 2023.

Move WS-Soap methods to REST service

Before version 2023, methods for the DKjord database were exposed through both a WS-Soap and REST endpoint. The WS-Soap endpoint was mainly used for writing data to the database, while the REST endpoint was used for extracting data from the database. Version 2023 retires the WS-Soap endpoint and moves all methods from it to the REST service.

The following methods were moved from WS-Soap to REST:

  • Create location
  • Update location
  • Delete location
  • Is alive
  • Generate soil contamination report

Changed endpoints

All endpoints will also change as the changes are so large, we saw it as simpler to have both the old and new systems running simultaneously.

Change DKJord to use OAuth

With the move of write methods to REST it was also possible to update security in a number of ways:

  • Update the REST API to use Oauth.
  • Force 3rd party systems to connect to endpoint using Codeflow integrated with DMP Identity and Access Management system.
  • All REST authorization is done using roles of the DMP Identity and Access Management system.
  • Update the Web App to run TLS 1.2 or higher for dkjord-api.

Split up methods for Location and Parcel

In prior versions, there were a lot of weaknesses in the way that DKjord handled Parcels:

  • Parcels were child objects of Location. As Parcels can actually belong to more than one Location, this created an inconsistency in the data model and potentially unclean data, given that all locations related to a changed Parcel were not updated at the same time.
  • Nuance value was not stamped onto the Parcel, so the responsibility of analyzing the correct value fell onto viewers analyzing maps with overlays showing soil pollution.
  • Extracts of parcels were offered to two parties (SDFE and ENET) using a very complex, performance-heavy and inefficient async process, that also delivered data in a format that was hard to analyze.
  • The analysis and conclusions for Parcel Pollution and Nuance status was not clearly done by the Regions, but fell indirectly unto DMP and its data consumers rather than placing governance and timestamping of this with the Regions.
  • DKjord was made responsible to figure out Region and Municipality of Parcels, even when Region systems already have this information.

To manage these problems, the following extensive changes were done:

  • Parcel creation and update was excluded from Location methods
  • Parcels were made distinct objects
  • Parcel methods for update, create, and delete were added.
  • Parcel objects were extended with Nuance value, decision time stamp, parcel relation to pollution status declarations, region code and municipality code. Basically moving governance of Parcel status to the Regions and avoiding any inconsistencies to end users.
  • Extensive Parcel search methods were added that could completely replace the need for SDFE and ENET extracts, meaning that third parties can now get exactly the data they need without needing to analyze and interpret these data.
  • Parcel Search method supports a number of query parameters making it easy to find and understand Parcels, like cadastralDistrictIdentifier, landParcelIdentifier, locationReference, regionCode, municipalityCode, pollutionStatus, pollutionNuanceStatus, includeGeometries, modifiedDateFrom, and recalculationDateFrom. This basically gives third parties a way to get the exact Parcels they need; even to keep data in sync without excessive data exchange.
  • Offer method for extracting historical Parcel values by date
  • Offer choice of include/exclude Parcels when searching for Locations.

These changes required a considerable change in the data model and also pushed a lot of responsibility back on the Regions' systems, that mean they had to:

  • Make sure to trigger a sending of Parcel object when affecting Locations were also sent.
  • Making sure to stamp both Pollution status, Nuance Status, array of affecting Status Declaration (pollution areas) references and calculation Date on Parcels (as well as implement the decision process for adding these values).
  • Resend all data from their systems for both locations and parcels. Basically a complete redo of sending all data to DKjord.

Geoserver Endpoints were NOT changed

The following page describes each layer as well as its attributes.

WFS/WMS endpoints are: UDV:

TEST:

DEMO:

PROD:

Layer: LokaliseretFlader

Displays: All soil contamination instances with related polygon in the system given state "Lokaliseret (uafklaret)" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided

Layer: NuanceringsFlader

Displays: All soil contamination instances with related polygon in the system given a nuance status, either "F0 Nuanceret", "F1 Nuanceret", or "F2 Nuanceret" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided
  • Nuanceringsstatus: The nuance statys given to this soil contamination area

Layer: UEKFlader

Displays: All soil contamination instances with related polygon in the system given state "Udgået efter kortlægning" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided

Layer: UIKFlader

Displays: All soil contamination instances with related polygon in the system given state "Udgået inden kortlægning" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided

Layer: V1Flader

Displays: All soil contamination instances with related polygon in the system given state "Jordforurening V1" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided
  • Jordforureningsattester: A semicolon separated list of links for the soilcontamination reports of the parcels listed in "Lokalitetsmatrikler"

Layer: V2Flader

Displays: All soil contamination instances with related polygon in the system given state "Jordforurening V2" Object: One per distinct combination of Lokalitetsnr, Fladegeometri and Lokalitetsejerlavkode Notice: This means that there for each polygon may be multiple records if polygon crosses over more Lokalitetsejerlavkode areas

Attributes:

  • ID: Distinct identifier
  • SenesteInddateringsdato: Datetime the record was last added/updated
  • Lokalitetsnr: Location number
  • Lokalitetsforureningsstatus: The contamination level set for the entire location
  • Lokalitetsadresse: The address of the location
  • Regionsnavn: The region of the locality
  • Lokalitetsejerlavkode: The "elav" information of parcel
  • Lokalitetsmatrikler: Semicolon seperated list of "Matrikelnumre" information of parcel
  • Fladegeometri: The polygon that the soil contamination area covers
  • Kortlaegningsdato: The date and time that the soil contamination status of this area was decided
  • Jordforureningsattester: A semicolon separated list of links for the soilcontamination reports of the parcels listed in "Lokalitetsmatrikler"

Clean up of the database and location object for unused tables and objects

As we were doing major changes anyway, it provided the opportunity to clean up the database and location methods the following places:

  • Remove table and object [AreaClassification] (not in use)
  • Remove table and object [JustificationClassification] (not in use)
  • Remove table [SectionAuthorizationXGeometry] (not in use)
  • Remove table [PollutionComponentXStuffGroup] (not in use)
  • Remove table [StuffGroup] (not in use)
  • Remove table and object [CostInCalendarYearAmount] (not in use)
  • Remove column and attribute [StatusDeclaration].[AreaMeasure] (requested by Regions to be retired)
  • Remove column and attribute [dbo.Parcel].[PollutionPartial] (requested by Regions to be retired)
  • Remove table [AreaClassificationRegulative] (not in use)
  • Remove table [DAIThemes] (not in use)
  • Remove table [Institution] (not in use)
  • Remove table [SYS_AdminLog] (not in use)
  • Remove table [SYS_CadastralDistricts] (not in use)
  • Remove table [SYS_DaiDeletions] (not in use)
  • Remove table [SYS_Events] (not in use)
  • Remove table [SYS_ExportJobs] (not in use)
  • Remove table [SYS_KmsAsyncResult] (not in use)
  • Remove table [SYS_KmsErrorLog] (not in use)
  • Remove table [SYS_KmsOutputQueue] (not in use)
  • Remove table [SYS_KmsOutputQueueItem] (not in use)
  • Remove table [SYS_KmsRegionResult] (not in use)
  • Remove table [SYS_KmsResponse] (not in use)
  • Remove table [SYS_KmsResponseItem] (not in use)
  • Remove table [SYS_Settings] (not in use)
  • Remove table [SYS_SimpleLog] (not in use)
  • Remove table [SYS_TransactionLogBackup] (not in use)

A few of the above tables had some data, but as we will take a backup and archive the old database, we will not lose any audit data from removing these tables from the new database version.

Improvements for soil contamination reports

The improvements made, especially for the Parcel object and process, has also meant that we have been able to make a lot of improvements for the soil contamination report:

  • Nuance status can now specifically be read on the report rather than assumed from the map overlay
  • Remove unused sections related to [AreaClassification] and [JustificationClassification]
  • Change map and parcel info provider from Kortforsyningen to Dawa and Datafordeler

Finally, we improved usability through a number of improvements:

  • Extensive documentation on Github: https://github.com/danmarksmiljoeportal/DkJord
  • Implementation of Swagger for REST endpoint
  • Addition of OpenAPI for documentation of REST endpoint
  • Set up Application Insights logging for the system