Skip to content

GSIP 34

Jump to bottom
Jody Garnett edited this page Jul 12, 2017 · 1 revision

GSIP 34 - New data directory structure for 2.x

Overview

A restructuring of the geoserver data directory for the 2.x series.

Motivation Proposal Backwards Compatability Feedback Voting Links

Proposed By

Justin Deoliveira

Assigned to Release

2.0

State

Under Discussion, In Progress, Completed, Rejected, Deferred

Motivation

One of the big drivers for moving to a new catalog and configuration api has been to replace the current XML persistence mechanism. The current mechanism has a couple of primary flaws:

  • It requires a change to the xml readers and writers every time a change occurs in a catalog or config object
  • It requires that all configuration be saved only when a single object changes

The first problem has been addressed by adopting the use of the XStream library which can automatically persist objects and vice versa. The second requires a change to how we store catalog and configuration inside of the data directory.

The primary motivation of this proposal is to come up with a data directory structure which:

  1. Better reflects the new catalog and configuration apis
  2. Allows objects to be persisted in a granular way

Proposal

The current data directory structure:

catalog.xml
services.xml
featureTypes/
   ft1/
     info.xml
     title.ftl
     description.ftl
   ft2/
     info.xml
   ...
coverages/
   c1/
     info.xml
   c2/
     info.xml
   ...
styles/
   s1.sld
   s2.sld
   ...
templates/

The proposed data directory structure:

global.xml
logging.xml
wms.xml
wcs.xml
wfs.xml
workspaces/
   ws1/
     ds1/
       datastore.xml
       ft1/
         featuretype.xml
         layer.xml
         title.ftl
         description.ftl
       ft2/
         featuretype.xml
         layer.xml
       ...
     ds2/
       datastore.xml
       ft1/
         featuretype.xml
         layer.xml
     ...
    ws2/
      cs1/
        coveragestore.xml
        c1/
          coverage.xml
          layer.xml
        ...
      cs2/
        coveragestore.xml
        c1/
          coverage.xml
          layer.xml
      ...
namespaces/
    topp.xml
    tiger.xml
    ...
layergroups/
    lg1.xml
    lg2.xml
styles/
    s1.sld
    s1.xml
    s2.sld
    s2.xml
    ...
templates/
    ...

(!) Things to note:

  • There is no catalog.xml, instead the catalog is inferred from the file system structure under the workspaces, namespaces, styles, and layergroups directories.
  • Datastores are not longer described in a single file. Each directory under a workspace directory contains a datastore.xml file which describes the data store.
  • Feature type directories are now located under the data store directory they correspond to. Feature types are described by featuretype.xml, rather than info.xml
  • Same story as 2 and 3 for coverage stores and coverages
  • Publishing information, like styles, for feature types and coverages is now persisted in the layer.xml located in the same directory. (!) When the resources and publishing are decoupled this will likely change somewhat, but should be able to support in a backwards compatible way.
  • There is no services.xml, instead the service configuration is split up into multiple files. global.xml stores service wide global configuration such as contact into, jai configuration, etc… Each service gets its own configuration file (wfs.xml,wcs.xml, etc…).
  • Logging configuration has been broken out into a separate file, logging.xml. This is mainly to better handle the “chicken and egg” nature of logging in which logging must be configured first thing at server start, however it is defined as part of the configuration, and before configuration is read logging must be configured.

File contents

global.xml

<global>
 <contact>
 <addressCity>Alexandria</addressCity>
 <addressCountry>Egypt</addressCountry>
 <addressType>Work</addressType>

<contactEmail>[claudius.ptolomaeus@gmail.com](mailto:claudius.ptolomaeus@gmail.com)</contactEmail>
 <contactOrganization>The ancient geographes INC</contactOrganization>
 <contactPerson>Claudius Ptolomaeus</contactPerson>
 <contactPosition>Chief geographer</contactPosition>
 </contact>
 <jai>
 <allowInterpolation>false</allowInterpolation>
 <recycling>false</recycling>
 <tilePriority>5</tilePriority>
 <tileThreads>7</tileThreads>
 <memoryCapacity>0.5</memoryCapacity>
 <memoryThreshold>0.75</memoryThreshold>
 <imageIOCache>false</imageIOCache>
 <pngAcceleration>true</pngAcceleration>
 <jpegAcceleration>true</jpegAcceleration>
 <allowNativeMosaic>false</allowNativeMosaic>
 </jai>
 <charset>UTF–8</charset>
 <numDecimals>8</numDecimals>

<onlineResource>[http://geoserver.org](http://geoserver.org)</onlineResource>
 <verbose>false</verbose>
 <verboseExceptions>false</verboseExceptions>
 <updateSequence>57</updateSequence>
</global>

h4. logging.xml

    <logging>
      <level>DEFAULT_LOGGING.properties</level>
      <location>logs/geoserver.log</location>
      <stdOutLogging>true</stdOutLogging>
    </logging>

wfs.xml

<wfs>
 <id>wfs</id>
 <enabled>true</enabled>
 <name>WFS</name>

<title>
GeoServer Web Feature Service

</title>
<maintainer>[http://jira.codehaus.org/secure/BrowseProject.jspa?id=10311](http://jira.codehaus.org/secure/BrowseProject.jspa?id=10311)</maintainer>
 <abstrct>This is the reference implementation of WFS 1.0.0 and WFS
1.1.0, supports all WFS operations including Transaction.</abstrct>
 <accessConstraints>NONE</accessConstraints>
 <fees>NONE</fees>
 <keywords>
 <string>WFS</string>
 <string>WMS</string>
 <string>GEOSERVER</string>
 </keywords>
 <metadataLink/>
 <citeCompliant>false</citeCompliant>

<onlineResource>[http://geoserver.sourceforge.net/html/index.php](http://geoserver.sourceforge.net/html/index.php)</onlineResource>

<schemaBaseURL>[http://schemas.opengis.net](http://schemas.opengis.net)</schemaBaseURL>
 <verbose>false</verbose>
 <gml>
 <entry>

<org.geoserver.wfs.WFSInfo_-Version>V*10</org.geoserver.wfs.WFSInfo_-Version>
 <org.geoserver.wfs.GMLInfoImpl>
 <srsNameStyle>XML</srsNameStyle>
 </org.geoserver.wfs.GMLInfoImpl>
 </entry>
 <entry>

<org.geoserver.wfs.WFSInfo_-Version>V*11</org.geoserver.wfs.WFSInfo_-Version>
 <org.geoserver.wfs.GMLInfoImpl>
 <srsNameStyle>URN</srsNameStyle>
 </org.geoserver.wfs.GMLInfoImpl>
 </entry>
 </gml>
 <serviceLevel>COMPLETE</serviceLevel>
 <maxFeatures>1000000</maxFeatures>
 <featureBounding>false</featureBounding>
</wfs>

h4. datastore.xml

<dataStore>
  <name>states_shapefile</name>
  <enabled>true</enabled>
  <workspace>
    <name>topp</name>
  </workspace>
  <connectionParameters>
    <namespace>
      <string>http://www.openplans.org/topp</string>
    </namespace>
    <url>
      <string>file:data/shapefiles/states.shp</string>
    </url>
  </connectionParameters>
</dataStore>

featuretype.xml

<featureType>
 <name>states</name>
 <nativeName>states</nativeName>
 <namespace>
 <name>topp</name>
 </namespace>

<title>
USA Population

</title>
<abstract>This is some census data on the states.</abstract>
 <keywords>
 <string>census</string>
 <string>united</string>
 <string>boundaries</string>
 <string>state</string>
 <string>states</string>
 </keywords>
 <nativeCRS>GEOGCS[“GCS*WGS*1984”, 
 DATUM[&quot;WGS*1984“, 
 SPHEROID[”WGS*1984“, 6378137.0, 298.257223563]] , 
 PRIMEM[”Greenwich“, 0.0], 
 UNIT[”degree“, 0.017453292519943295], 
 AXIS[”Longitude“, EAST], 
 AXIS[”Latitude&quot;, NORTH]] </nativeCRS>
 <srs>EPSG:4326</srs>
 <nativeBoundingBox>
 <minx>~~124.73142200000001</minx>
 <maxx>~~66.969849</maxx>
 <miny>24.955967</miny>
 <maxy>49.371735</maxy>
 <crs>EPSG:4326</crs>
 </nativeBoundingBox>
 <latLonBoundingBox>
 <minx>~~124.731422</minx>
 <maxx>~~66.969849</maxx>
 <miny>24.955967</miny>
 <maxy>49.371735</maxy>
 <crs>EPSG:4326</crs>
 </latLonBoundingBox>
 <projectionPolicy>FORCE*DECLARED</projectionPolicy>
 <enabled>true</enabled>
 <metadata>
 <cacheAgeMax>
 <string>3600</string>
 </cacheAgeMax>
 <kml.regionateFeatureLimit>
 <int>10</int>
 </kml.regionateFeatureLimit>
 <indexingEnabled>
 <boolean>false</boolean>
 </indexingEnabled>
 <cachingEnabled>
 <boolean>true</boolean>
 </cachingEnabled>
 <dirName>
 <string>states</string>
 </dirName>
 </metadata>
 <store class="dataStore">
 <name>states*shapefile</name>
 </store>
 <attributes>
 <attribute>
 <name>the*geom</name>
 <minOccurs>0</minOccurs>
 <maxOccurs>1</maxOccurs>
 <nillable>false</nillable>
 </attribute>
 <attribute>
 <name>STATE*NAME</name>
 <minOccurs>0</minOccurs>
 <maxOccurs>1</maxOccurs>
 <nillable>false</nillable>
 </attribute>
 …
 <maxFeatures>0</maxFeatures>
 <numDecimals>0</numDecimals>
</featureType>

h4. coveragestore.xml

<coverageStore>
  <name>arcGridSample</name>
  <enabled>true</enabled>
  <workspace>
    <name>nurc</name>
  </workspace>
  <type>ArcGrid</type>
  <url>file:coverages/arc_sample/precip30min.asc</url>
</coverageStore>

coverage.xml

<coverage>
  <name>Arc_Sample</name>
  <nativeName>Arc_Sample</nativeName>
  <namespace>
    <name>nurc</name>
  </namespace>
  <title>Global annual rainfall</title>
  <description>Global annual rainfall in ArcGrid format</description>
  <keywords>
    <string>WCS</string>
    <string>arcGridSample</string>
    <string>arcGridSample_Coverage</string>
  </keywords>
  <nativeCRS>GEOGCS[&quot;WGS 84&quot;, 
  DATUM[&quot;World Geodetic System 1984&quot;, 
    SPHEROID[&quot;WGS 84&quot;, 6378137.0, 298.257223563, AUTHORITY[&quot;EPSG&quot;,&quot;7030&quot;]] , 
    AUTHORITY[&quot;EPSG&quot;,&quot;6326&quot;]] , 
  PRIMEM[&quot;Greenwich&quot;, 0.0, AUTHORITY[&quot;EPSG&quot;,&quot;8901&quot;]] , 
  UNIT[&quot;degree&quot;, 0.017453292519943295], 
  AXIS[&quot;Geodetic longitude&quot;, EAST], 
  AXIS[&quot;Geodetic latitude&quot;, NORTH], 
  AUTHORITY[&quot;EPSG&quot;,&quot;4326&quot;]] </nativeCRS>
  <srs>EPSG:4326</srs>
  <nativeBoundingBox>
    <minx>-180.0</minx>
    <maxx>180.0</maxx>
    <miny>-90.0</miny>
    <maxy>90.0</maxy>
    <crs>EPSG:4326</crs>
  </nativeBoundingBox>
  <latLonBoundingBox>
    <minx>-180.0</minx>
    <maxx>180.0</maxx>
    <miny>-90.0</miny>
    <maxy>90.0</maxy>
    <crs>EPSG:4326</crs>
  </latLonBoundingBox>
  <enabled>true</enabled>
  <metadata>
    <dirName>
      <string>arcGridSample_Arc_Sample</string>
    </dirName>
  </metadata>
  <store class="coverageStore">
    <name>arcGridSample</name>
  </store>
  <nativeFormat>ArcGrid</nativeFormat>
  <grid dimension="2">
    <range>
      <low>0 0</low>
      <high>720 360</high>
    </range>
    <transform>
      <scaleX>0.5</scaleX>
      <scaleY>-0.5</scaleY>
      <shearX>0.0</shearX>
      <shearX>0.0</shearX>
      <translateX>-179.75</translateX>
      <translateY>89.75</translateY>
    </transform>
    <crs>EPSG:4326</crs>
  </grid>
  <supportedFormats>
    <string>ARCGRID</string>
    <string>ARCGRID-GZIP</string>
    <string>GEOTIFF</string>
    <string>PNG</string>
    <string>GIF</string>
    <string>TIFF</string>
  </supportedFormats>
  <interpolationMethods>
    <string>nearest neighbor</string>
  </interpolationMethods>
  <defaultInterpolationMethod>nearest neighbor</defaultInterpolationMethod>
  <dimensions>
    <coverageDimension>
      <name>precip30min</name>
      <description>GridSampleDimension[-9999.0,8849.000000000002]</description>
      <range>
        <min>-9999.0</min>
        <max>8849.0</max>
      </range>
    </coverageDimension>
  </dimensions>
  <requestSRS>
    <string>EPSG:4326</string>
  </requestSRS>
  <responseSRS>
    <string>EPSG:4326</string>
  </responseSRS>
</coverage>

layer.xml

<layer>
 <name>states</name>
 <path>/</path>
 <type>VECTOR</type>
 <defaultStyle>
 <name>population</name>
 </defaultStyle>
 <styles>

<style>
<name>polygon</name>

</style>

<style>
<name>pophatch</name>

</style>
</styles>
 <resource class="featureType">
 <name>states</name>
 </resource>
 <enabled>true</enabled>
</layer>

Persisting with events

As mentioned above, in GeoServer 1.x persisting a single change means persisting every single configuration object. In 2.x changes are persisted in response to events. The catalog and configuration apis provide a listener api. This api is used to listen to catalog and configuration events, and persist them out as they occur. The process is as follows:

*. On server start up after the catalog is read, a listener is attached to it. Similar for the configuration. This happens in the GEoServerLoader class *. Every time an object is changed (ie any of the Catalog.save () or GeoServer.save () methods are called), an event is fired. The listener takes the object and uses xstream to persist the change.

With this system when only a single object is changed, only a single file is written out.

Importing 1.x style data directories

1.x style directories are handled transparently on startup. When the server starts up a check for the existance of catalog.xml made. If it exists a 1.x data directory is assumed and an import occurs. The import is performed by the LegacyCatlogImporter class.

During the import the data directory is transformed into a new style data directory. For example when a 1.x style feature type directory is read, a 2.x style directory is created in parallel. After the import is comlete the catalog.xml file is removed/backed up/renamed so that it will be ignored on subsequent start ups. An identical process is performed for services.xml.

It is also planned to provide an interactive utility from the user interface for importing old style data directories. Doing so is relatively simple and simply involves reusing the same classes used to do the import automatically on startup.

Feedback

Backwards Compatibility

This is a new data directory structure. However it has generally been known and accepted that come 2.x data directory structure will change. That said the old style data directories will still be support as an “import”.

Voting

Andrea Aime: +1 Alessio Fabiani Justin Deoliveira: +1 Jody Garnett: +1 Saul Farber: Rob Atkinson: +1

Links

JIRA Task Email Discussion Wiki Page

©2022 Open Source Geospatial Foundation

Home
Proposals
Release Schedule
Maintenance & Roadmap
Sponsor

Clone this wiki locally