Ecere GeoPackage Tiling Experiment 3 - sofwerx/cdb2-concept GitHub Wiki

CDB X Tiling/GeoPackage Experiments set #3 (Ecere)

In this third set of experiments, a 3D models were added to the GeoPackages, as well as a table of textures used by those models. Two variations were produced:

A) one where 3D models are individually stored in a single table, and referenced and placed by tiled vector data (points),
B) and another where batched 3D models are the content of the tiles.

Both variations, as well as the tiled elevation, imagery and vector layers produced in previous experiments, are available in the links provided below.

A) Referenced 3D models with placement information

In this approach, a single table per GeoPackage (gpkgext_3d_models) is used to define one model per row, in a blob within the model field. A format field allows to specify the format. Both glTF and E3D have been used in the experiments. The name field allows to specify a name for the model, andt the lod field can optionally be used to distinguish between multiple level of details for the model, or left NULL if only a single version exists. The combination of name, lod and format must must be unique.

The model::id field of the attributes table for the 3D models referencing points (which would also contain the point geometry in a non-tiled approach) references the id (primary key) of the gpkgext_3d_models table. The attributes table may also contain additional fields for scaling and orienting the models:

model::scale; or model::scaleX, model::scaleY and model::scaleZ for non-uniform scaling
model::yaw, model::pitch and model::roll

These attributes duplicate the CDB fields AO1, SCALx, SCALy, SCALz (and in a sense MODL as well), but are intended to be defined in a non-CDB specific manner within a generic 3D Models extension for GeoPackage.

For the case of points referencing the 3D models (best suited for geo-typical), those glTF files would be stored in a single 3D models table, as well as a textures table (if the models share rather than embed textures).

The following SQL is used to create the gpkgext_3d_models table:

CREATE TABLE gpkgext_3d_models (
   id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
   name TEXT NOT NULL,
   lod INTEGER,
   format TEXT NOT NULL,
   model BLOB,
   CONSTRAINT unique_models UNIQUE(name, lod, format));

Example gpkgext_3d_models:

id  name               lod  format  model
--  -----------------  ---  ------  -----
1   coniferous_tree01       glb     glTF
2   palm_tree01             glb     glTF

Sample SQL table creation for attributes table referencing the 3D models:

CREATE TABLE attributes_Trees (
   id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
   AO1 REAL, CNAM TEXT, RTAI INTEGER,
   SCALx REAL, SCALy REAL, SCALz REAL,
   AHGT TEXT, BBH REAL, BBL REAL, BBW REAL, BSR REAL,
   CMIX INTEGER, FACC TEXT, FSC INTEGER, HGT REAL,
   MODL TEXT,
   `model::id` INTEGER,
   `model::yaw` REAL,
   `model::scale` REAL)

B) Batched 3D models as tile payload

In this approach, a single model covers a whole tile, batching all 3D models from the data layer found within that tile, and is stored in a tiles table much like raster or vector tiles (as a glTF blob in the tile_data field). This approach works best for geo-specific models. It is closer to the 3D Tiles / One World Terrain approach, and could potentially also combine both 3D Terrain and 3D Models (though ideally keeping them as distinct nodes within the model). Such an approach may facilitate transition between CDBX and OWT.

Because GeoPackage does not define a generic mechanism to specify the encoding of tile_data (it has previously been suggested that this would be a good field to add to the gpkg_contents table), the encoding of the 3D model must be deducted from the content of the blob. Fortunately, both glTF and E3D feature a header signature which facilitates this. The 3d-models type is introduced to specify in the data_type column of the gpkg_contents table.

The translation origin of the model, as well as its orientation, is implied from the center of the tile (from the tile matrix / tile matrix set) for which it defined. The model is defined in the 3D cartesian space where (0, 0, 0) lies at that center, sitting directly on the WGS84 ellipsoid, and oriented so that by default it appears upright with its X axis pointing due East, its Z axis pointing North, and its Y axis pointing away from the center of the Earth. In other words, it would be equivalent to having a single point situated at the center of the tile in the referenced 3D points approach.

The height of the individual features (e.g. buildings) within the batched models tile models has already been adjusted match the elevation model. However, each separate feature from CDB is encoded in the model as a separate node to facilitate re-adjusting it to new elevation.

Textures table

The textures table has the following fields:

id (integer primary key)
name: The filename used in the model to refer to the texture
width: width of the texture
height: height of the texture
format: e.g. "png"
texture: blob containing the texture data

The combination of name, width, height and format must be unique.

The following SQL statement is used to create the table:

CREATE TABLE gpkgext_textures (
   id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
   name TEXT NOT NULL,
   width INTEGER NOT NULL,
   height INTEGER NOT NULL,
   format TEXT NOT NULL,
   texture BLOB,
   CONSTRAINT unique_textures UNIQUE(name, width, height, format));

id  name   width  height  format  texture
--  -----  -----  ------  ------  -------
1   1.png  512    512     png     �PNG
2   1.png  256    256     png     �PNG
3   2.png  512    512     png     �PNG
4   2.png  256    256     png     �PNG

3D Models Extension

As a draft GeoPackage extension for 3D models is being defined, rows are added to the gpkgext_extensions table to identify all tables set up for this extension.

For all of these tables, the extension_name is configured to be ecere_3d_models, the definition http://github.com/ecere/gpkgext-3dmodels, and the scope read-write.

The tables registered with this extension are:

gpkgext_3d_models (only used for the reference points approach A)
the individual tiles tables for batched 3D models (only used for approach B)
gpkgext_textures for shared textures

Data results

Single GeoPackage

This one GeoPackage containing the whole San Diego CDB data (not including Blue Marble):

Complete San Diego CDB X - cdb.json index

Complete San Diego CDB X - GeoPackage

Complete San Diego CDB X - GeoPackage using GNOSIS Map Tiles and E3D Models

(Links to be updated after available from OGC portal)

Data layers packaged together (tiles grouped in multiple GeoPackages)

This series of GeoPackage containing the whole San Diego CDB data (not including Blue Marble):

cdb.json index

One package using an LOD grouping of 5

Partial dataset (subset of overall extent):

Partial dataset using an LOD grouping of 5

As separate layers

Top-level cdb.json index

Elevation (encoded as 16-bit integer PNG)

Elevation (encoded as 32-bit float GeoTIFF)

Elevation (encoded as 16-bit integer Paeth/LZMA GNOSIS Map Tiles)

Imagery:

Full resolution global coverage (for one particular month) of the NASA Visible Earth Blue Marble

Medium resolution imagery

High resolution imagery

Vector data (tiled, using Mapbox Vector Tiles):

Hydrography

Roads

Airport Lights

Vector data (tiled, using GNOSIS Map Tiles):

Hydrography

Roads

Airport Lights

3D Models (reference points):

Buildings

Trees

Coronado Bridge

More compact GNOSIS Map Tiles / E3D versions of the referenced 3D models:

Buildings (GNOSIS Map Tiles / E3D)

Trees (GNOSIS Map Tiles / E3D)

Coronado Bridge (GNOSIS Map Tiles / E3D)

Different grouping of LODs (glTF):

Buildings grouped by 4 LODs

Buildings grouped by 5 LODs

Buildings grouped by 7 LODs

NOTE: Except for the directory / filename naming, the 7 LODs buildings are equivalent to 6 LODs since there are only 6 LODs of buildings in the dataset.

Trees grouped by 6 LODs

NOTE: Except for the directory / filename naming, the 6 LODs trees are equivalent to any other LOD groupings, since there is only 1 LOD of trees in the dataset.

Batched 3D Models tiles (glTF):

Buildings (glTF / batched 3D models tiles)

Trees (glTF / batched 3D models tiles)

Coronado Bridge (glTF / batched 3D models tiles)

Batched 3D Models tiles (E3D):

Buildings (E3D / batched 3D models tiles)

Trees (E3D / batched 3D models tiles)

Coronado Bridge (E3D / batched 3D models tiles)

OGC API access demo

At this address: https://maps.ecere.com/ogcapi/collections/SanDiegoCDB

the San Diego CDB data from the GNOSIS data store can be accessed directly through the GNOSIS Map Server, including rendering maps, downloading coverages, accessing as tiles in different tiling schemes, accessing individual vector features, retrieving them as (re-merged) GeoJSON, visualizing them on GeoJSON.io and so on. To some extent, this demonstrates that even though the data is tiled, this layout actually supports a wide range of use cases.

At this address: https://maps.ecere.com/ogcapi/collections/SanDiegoLayers

the prototype CDB X split GeoPackages can be accessed directly . However at this point, the map server and visualization client do not present this as a unified data source, so instead the tiles structure and individual GeoPackages are individually accessible.

Visualization

The GNOSIS visualization tools can currently visualize the individual CDB X/GeoPackage elevation and imagery directly. Like for the server however the split geopackages are not yet unified as a single data source. Accessing and visualizing the 3D models from the GeoPackage tables remains to be implemented.

Screenshots of visualization of the intermediate GNOSIS data store used to generate the CDB X using the same tiling scheme follow. Part of this effort was accomplished during the OGC Interoperable Simulation and Gaming Sprint. A video was also published.

In addition to the San Diego CDB dataset, worldwide elevation data from Viewfinder Panoramas by Jonathan de Ferranti and imagery from NASA Visible Earth’s Blue Marble are used outside of the extent covered by the San Diego dataset.

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (cape)

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (hotels and palm trees)

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (skyscrapers)

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (Coronado bridge)

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (airstrip)

San Diego CDB data visualized in Ecere’s GNOSIS Cartographer (high above, showing 3D globe)

This last image features ESA Gaia’s Sky in colour (Gaia Data Processing and Analysis Consortium (DPAC); A. Moitinho / A. F. Silva / M. Barros / C. Barata, University of Lisbon, Portugal; H. Savietto, Fork Research, Portugal.) CC BY SA 3.0.

Cesium JS / 3D Tiles demo

CesiumJS Client accessing San Diego CDB data as 3D Tiles from Ecere’s GNOSIS Map Server (Petco Park)

The following JavaScript code, which can simply be copied to the Cesium Sand Castle, can be used to visualize the data as 3D Tiles. It sets up the buildings, trees as well as the Coronado Bridge, together with the Cesium world terrain. This now includes textures. One limitation is that the generated tileset is still missing multiple level of details, therefore visualizing a large area will be quite slow.

var worldTerrain = Cesium.createWorldTerrain({ requestWaterMask: true, requestVertexNormals: true });
var viewer = new Cesium.Viewer("cesiumContainer", { terrainProvider: worldTerrain });
var scene = viewer.scene;
var trees = scene.primitives.add(new Cesium.Cesium3DTileset(
   { url: "https://maps.ecere.com/ogcapi/collections/SanDiegoCDB:Trees/3DTiles/tileset.json" }));
var bridge = scene.primitives.add(new Cesium.Cesium3DTileset(
   { url: "https://maps.ecere.com/ogcapi/collections/SanDiegoCDB:CoronadoBridge/3DTiles/tileset.json" }));
var buildings = scene.primitives.add(new Cesium.Cesium3DTileset(
   { url: "https://maps.ecere.com/ogcapi/collections/SanDiegoCDB:Buildings/3DTiles/tileset.json" }));

This 3D Tiles distribution is currently being generated from the GNOSIS Data Store / E3D models. Support to stream as 3D Tiles straight from CDBX GeoPackages should also be achievable.

Future work

Support for visualizing 3D models directly from the CDB X/GeoPackages dataset in GNOSIS Cartographer client
Support for GNOSIS Map Server streaming 3D models directly from CDB X/GeoPackage
Support for unifying split GeoPackages making up the CDB X dataset as a single data source
Attribution per model within the single tile model. This is supported directly in E3D for triangular face-level attribution (it was clarified that glTF2 does not supports this, and extensions were considering vertex rather than face attributions).