Rest file uploads put uploaded files in fixed locations inside the data directory. Roll an extension point to allow control of the target location

Andrea Aime

2.6.0

Choose one of: Under Discussion, In Progress, Completed, Rejected, Deferred

The RESTful API allows to create new stores and append new granules to mosaics via file uploads. The uploads currently use a fixed directory structure following this rule:

$GEOSERVER_DATA_DIR/data/<workspace>/<store>[/<file>]

In particular, all files end up in the data directory, and mosaics made of a large amount of files end up having a flat structure, which is detrimental to performance. The basic objectives of this proposal are two:

• Be able to store the files somewhere outside of the data directory, while still managing the upload via a REST API
• Allow plugins to control the location of the harvested files, in particular make it possible to create a community module that will spread out mosaic granules in subdirectories based on some internal rules (possibly regular expression based)

In order to make this general enough for GeoServer core inclusion we suggest a generic path mapping mechanism, inspired by how URLMangler can transform GeoServer backlinks, that can map a uploaded file to the desired path on the server.

Add a RESTUploadPathMapper interface that allows the implementors to change the location of the files, without giving up a bit of control to ensure files associated to different workspaces are going to be located in separate directories.

/**
 * Plugin interface used to transform the position of files during rest uploads
  */
interface RESTUploadPathMapper {
    /**
     * Remaps the position of a store path. The implementor is free 
     * to append, modify or replace the store root directory, REST 
     * upload will append workspace/store to it
     */
   void mapStorePath(StringBuilder rootDir, String workspace, String store, 
                     Map<String, String> storeParams);

    /**
     * Remaps the position of a file inside a store (e.g., a image 
     * being harvested into a mosaic. The implementor is free to alter 
     * both the root dir, and the item path
     */
   void mapItemPath(StringBuilder rootDir, String workspace, String store, 
                    Map<String, String> storeParams, StringBuilder itemPath);
}

The interface uses StringBuilder like URLMangler to allow path customization, the caller is free to append, modify or overwrite some bits of the paths. In particular:

• mapStorePath is used when creating a new store, and allows the base path to be altered, while providing information about the workspace and store being created (implementors might decide to act in a different way for a mosaic and a directory of shapefiles)
• mapItemPath is used when adding a new file in an existing file (harvest operation for structured grid readers at the moment, but we could have something similar for directory of shapefiles in the future for example) and allows also to control the location of the file inside the store directory

REST wise, the StoreFileResource doFileUpload/handleFileUpload methods will apply the RESTUploadPathMapper implementations found in the app context in sequence in order to get the final location of the store or of the file being uploaded.

In order to avoid cross workspace contamination, the final path will always be:

  ${rootDir}/workspace/store[/${itemPath}]

Where rootDir will be initially set to $GEOSERVER*DATA*DIR/data and itemPath, if present, to the file name.

Implementation wise we believe a single RootPathMapper will be added to core, which will be used to override the default root location based on configuration.

The configuration for the root directory will be added to the global settings as part of the metadata map (the REST config module being a plugin that people could decide to remove) and per workspace local settings will be able to override it.

In order to support this a web-rest module will be created, and an extension point SettingsPanel will be created, similar to how the service page can be extended via AdminPagePanel, to add the configuration of the root path (and allow other path mappers to have a UI in the same way).

A community module with regular expression based file position mapping will follow to exercise the API further and allow spreading the files in a mosaic in subdirectories based on their dimension values.

This section should contain feedback provided by PSC members who may have a problem with the proposal.

• geoserver-devel discussion

State here any backwards compatibility issues.

Alessio Fabiani: +1 Andrea Aime: +1 Ben Caradoc Davies: +1 Christian Mueller: Gabriel Roldan: +0 Jody Garnett: +1 Jukka Rahkonen: +0 Justin Deoliveira: +1 Phil Scadden: Simone Giannecchini: +0

JIRA Task Email Discussion Wiki Page