ESRI Shapefile - ObjectVision/GeoDMS GitHub Wiki
ESRI Shapefiles are used to store vector data and related attribute data.
The ESRI Shapefile format always consists of (at least) three accompanying files, with the extensions:
- .shp: containing the geographical coordinates (feature attribute).
- .shx: containing a spatial index.
- .dbf: containing tabular information describing the shapes. If a logical entity (like countries) consists of multiple shapes, the relation between the shapes and the logical entity is usually stored in this .dbf file.
Optionally a .prj file is also available, containing the projection information. This file is at the moment not (yet) used by the GeoDMS (projection information is configured explicitly in the GeoDMS).
The GeoDMS supports at the moment (multi)point, arc and polygon types (Shape types: 1,3, 5 and 8).
The GeoDMS supports two ways of reading ESRI Shapefiles:
- gdal.vect: for most shape files, we advise to use gdal.vect (see next subparagraph) to read these files as it:
- is faster
- is more generic
- is shorter to configure
- supports open options
- supports segmented data (If the file has more than 50.000 records, tiled domain with segments of maximum 50.000 entries will be made)
- shapefile/dbf StorageManager: the GeoDMS specific ESRI shapefile storage manager, which keeps the .shp and the .dbf as two independent storages.
how to configure reading point, polygon and arc data with gdal.vect:
unit<uint32> location
: StorageName = "%projDir%/vectordata/points.shp"
, StorageType = "gdal.vect"
{
attribute<point_rd> geometry;
attribute<string> name;
}
unit<uint32> region
: StorageName = "%projDir%/vectordata/region.shp"
, StorageType = "gdal.vect"
{
attribute<point_rd> geometry (polygon);
attribute<string> name;
}
unit<uint32> road
: StorageName = "%projDir%/vectordata/road.shp"
, StorageType = "gdal.vect"
{
attribute<point_rd> geometry (arc);
attribute<string> name;
}
The .shp file is configured as StorageName for the domain unit.
The .dbf and .shx files are not explicitly configured, but need to be available in the same folder as the .shp file.
The difference between the configuration of points, arcs and polygons is the configured composition type.
If your dbf file already contains an attribute named geometry, use another attribute name for the data from the .shp file, for instance geom.
GDAL options can be configured for reading different ESRI Shapefiles.
See: https://gdal.org/drivers/vector/shapefile.html#open-options for a full list of all open options.
Shapefiles can also still be read with the GeoDMS specific shapefile/dbf StorageManagers (mainly for backward compatibility).
how to configure reading point, polygon and arc data with shapefile/dbf StorageManager:
unit<uint32> location: StorageName = "%projDir%/data/location.dbf"
{
attribute<point_rd> geometry: StorageName = "%projDir%/data/location.shp", StorageType = "shp";
attribute<string> name;
}
unit<uint32> region: StorageName = "%projDir%/data/region.dbf"
{
attribute<point_rd> region (polygon): StorageName = "%projDir%/data/region.shp", StorageType = "shp";
attribute<string> name;
}
unit<uint32> road: StorageName = "%projDir%/data/road.dbf"
{
attribute<point_rd> geometry (arc): StorageName = "%projDir%/data/road.shp", StorageType = "shp";
attribute<string> name;
}
The dbf file is configured as StorageName of the domain unit. The number of elements is read from this file.
The feature attribute with the coordinates is configured as subitem of the domain unit. This attribute is read from the explicitly configured .shp file. The .shx file is not configured, the file needs to be available in the same folder as the .shp file.
Since GeoDMS 17.0.0 the StorageType has to be spelled out. Before that, a StorageName ending in .shp selected this storage manager on its extension; the extension now maps to gdal.vect instead. Reading mostly still succeeds through GDAL, so a configuration without a StorageType keeps working, but it is then read by a stricter reader: GDAL refuses a polygon whose rings are not closed, with Check Failed Error: ring->getY(0) == ring->getY(numPoints-1), where this storage manager reads such a ring and closes it itself.
A shapefile the GeoDMS itself wrote before 20.19.2 can be exactly such a file, because both writers stored the rings of a polygon as the value held them — see polygon rings are closed on write under Write below. Reading such an older file back needs StorageType = "shp", or rewriting it with 20.19.2 or later.
The difference between the configuration of points, arcs and polygons is the configured composition type.
A shapefile without its .dbf is not a complete shapefile: QGIS opens one, ArcGIS Pro refuses
it. If an export produced no .dbf, see Export primary data — two defects that caused
exactly that were fixed in GeoDMS 20.18.0.
The GeoDMS supports two ways of writing ESRI Shapefiles:
- gdalwrite.vect: for most shape files, we advise to use gdalwrite.vect (see next subparagraph) to write these files as it:
- is faster
- is more generic
- is shorter to configure
- supports open options
- shapefile/dbf StorageManager: for the cases gdalwrite.vect cannot express, since it writes one layer in which every record has one geometry, as GDAL and ESRI require. This storage manager writes the .shp and the .dbf as two independent storages, so the feature attribute and the tabular attributes need not be one on one related.
Since GeoDMS 20.19.2 both writers close the rings of a polygon as the ESRI specification requires: where the value does not repeat a ring's first point at the end, the writer appends it. This applies to gdalwrite.vect and to the shapefile/dbf storage manager alike, and it is the only place where the written coordinates differ from the ones the feature attribute holds.
Before 20.19.2 the rings were stored exactly as held, so a value with an unclosed ring produced a file that violates the specification and that the GeoDMS's own default reader refuses:
gdal Warning(1): Non closed ring detected. To avoid accepting it, set the
OGR_GEOMETRY_ACCEPT_UNCLOSED_RING configuration option to NO
Check Failed Error: ring->getY(0) == ring->getY(numPoints-1)
Only StorageType = "shp" read such a file back, because that reader closes a ring itself, so the
defect stayed invisible in a configuration that wrote and re-read with the same storage manager and
surfaced the moment the file went to gdal.vect, QGIS or ArcGIS. Note that a value can only be
unclosed when the polygon has a single ring: a multi-ring value delimits its rings by a point
that repeats the ring's own first point, so its rings are closed by construction. Files written by an
older version stay as they are; rewrite them, or read them with StorageType = "shp".
Closing at write time does not repair a point sequence that is wrong in other ways. Winding order is not touched here — see Point order in polygons and fix_winding_order.
how to configure writing point, polygon and arc data with gdalwrite.vect:
unit<uint32> location := SourceData/location
, StorageName = "%LocalDataProjDir%/export/points.shp"
, StorageType = "gdalwrite.vect"
{
attribute<point_rd> geometry := SourceData/location/geometry;
attribute<string> name := SourceData/location/name;
}
unit<uint32> region := SourceData/region
, StorageName = "%projDir%/vectordata/region.shp"
, StorageType = "gdalwrite.vect"
{
attribute<point_rd> geometry (polygon) := SourceData/region/geometry;
attribute<string> name := SourceData/region/name;
}
unit<uint32> road := SourceData/road
, StorageName = "%projDir%/vectordata/road.shp"
, StorageType = "gdalwrite.vect"
{
attribute<point_rd> geometry (arc) := SourceData/road/geometry;
attribute<string> name := SourceData/road/name;
}
The feature attribute is written to the .shp file (in the examples geometry).
All other attributes (with supported .dbf value types) are written to a filename with the same name and a .dbf extension. This applies to both the direct and the indirect subitems. If multiple subitems with a PointGroup value type occur, an error is generated. Set the DisableStorage property to True for the attributes you do not want to be written to the.dbf file.
An index file is also written together with the .shp file, with the same name but with the .shx extension.
The difference between the configuration of points, arcs and polygons is the configured composition type.
GDAL options can be configured for writing different ESRI Shapefiles.
See: https://gdal.org/drivers/vector/shapefile.html#layer-creation-options for a full list of all creation options.
example on how to configure a write option:
container with_index
{
unit<uint32> optionSet := range(uint32, 0, 1);
attribute<string> GDAL_LayerCreationOptions (optionSet) : ["SPATIAL_INDEX=YES"];
unit<uint32> location := SourceData/location
, StorageName = "%LocalDataProjDir%/export/points.shp"
, StorageType = "gdalwrite.vect"
{
attribute<point_rd> geometry := SourceData/location/geometry;
attribute<string> name := SourceData/location/name;
}
}
This example shows how to export an ESRI Shapefile with a spatial index file (.qix extension).
Shapefiles can also still be written with the GeoDMS specific shapefile/dbf StorageManagers. Use them where the feature attribute and the tabular attributes are not one on one related, which is what gdalwrite.vect (and ESRI) require: here the .shp and the .dbf are two independent storages, each configured on its own item.
how to configure writing point, polygon and arc data with shapefile/dbf StorageManager:
container export
{
unit<uint32> location := SourceData/location, StorageName = "%projDir%/data/location.dbf"
{
attribute<point_rd> geometry := SourceData/Location/geometry
, StorageName = "%projDir%/data/location.shp", StorageType = "shp";
attribute<string> name := SourceData/Location/name;
}
unit<uint32> region := SourceData/region, StorageName = "%projDir%/data/region.dbf"
{
attribute<point_rd> region (polygon) := SourceData/region/geometry
, StorageName = "%projDir%/data/region.shp", StorageType = "shp";
attribute<string> name := SourceData/region/name;
}
unit<uint32> road := SourceData/road, StorageName = "%projDir%/data/road.dbf"
{
attribute<point_rd> geometry (arc) := SourceData/road/geometry
, StorageName = "%projDir%/data/road.shp", StorageType = "shp";
attribute<string> name := SourceData/road/name;
}
}
The dbf file is configured as StorageName of the domain unit. The attributes (with supported .dbf value types) are written to the .dbf file with this filename.
The feature attribute is written to the explicitly configured .shp file. An index file is also written together with it, with the same name but with the .shx extension.
Since GeoDMS 17.0.0 the StorageType = "shp" is required here; without it the run stops on the feature attribute and no .shp file is written. Before that, a StorageName ending in .shp selected this storage manager on its extension; the extension now maps to gdal.vect, which cannot write, so the run stops on the feature attribute with Item has both a Calculation Rule and a read-only storage spec.
Do not configure StorageType = "gdalwrite.vect" on the feature attribute instead. That writes a whole layer, .dbf included, over the .dbf that the domain unit writes, and which of the two survives depends on the order in which they run.
The feature attribute above is called region, not geometry: this storage manager writes the item that carries the .shp StorageName, whatever its name. The name rule of gdalwrite.vect, where the geometry is recognized by its composition since GeoDMS 20.19.2, applies there and not here.
GDAL options have no effect on this storage manager, which reads none of them: a .qix spatial index and RESIZE are options of the GDAL ESRI Shapefile driver, see the gdalwrite.vect subparagraph above.
The difference between the configuration of points, arcs and polygons is the configured composition type.