Test Suite Strawman - opengeospatial/developer-events GitHub Wiki

Test Matrix

Sample

Test CubeWerx (CW) CRIM (CR) Ecere (EC) GeoLabs (GL)
1 :white_check_mark: (CR), :white_check_mark: (EC), :white_check_mark: (GL) :white_check_mark: (CW), :white_check_mark: (EC), :white_check_mark: (GL) :white_check_mark: (CW), :white_check_mark: (CR), :white_check_mark: (GL) :white_check_mark: (CW), :white_check_mark: (CR), :white_check_mark: (EC)
2 ❌ (CR), ❌ (EC), ❌ (GL) ❌ (CR), ❌ (EC), ❌ (GL) ❌ (CW), ❌ (CR), ❌ (GL) ❌ (CW), ❌ (CR), :no_entry_sign: (EC)

Legend:

  • ✅ - test passed
  • ❌ - test failed
  • 🚫 - capability not implemented by server
  • ❔ - not yet tested
  • 🚧 - test under implementation
  • ⚠️ - test mostly working except a condition/case (see detail in sub-table)

The value in parentheses after the icon indicates who executed the test. The codes are:

  • CW - CubeWerx
  • CR - CRIM
  • EC - Ecere
  • GL - GeoLabs

OGC API - Processes - Part 1: Core

Test CubeWerx (CW) CRIM (CR) Ecere (EC) GeoLabs (GL)
1. Landing Page ✅(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ✅(CR) ❔(GL) ❔(CW) ✅(CR) ❔(EC)
2. Conformance Page ✅(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❌(CR) ❔(GL) ❔(CW) ❌(CR) ❔(EC)
3. API Definition 🚫(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❌(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
4. Process List ✅(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
5. Process Description ❌(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
6. Process Execution ⚠️(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
7. Job Status ✅(CR) ❔(EC) ❔(GL) ✅(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
8. Job Results ⚠️(CR) ❔(EC) ❔(GL) ⚠️(CR) ✅(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
8d. Job Result ✅(CR) ❔(EC) ❔(GL) ❌(CR) ❌(CW) ❔(EC) ❔(GL) ❔(CW) ❔(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)

Notes

Test Tested Tester Note
1 GL CR Request https://ogcapip-cs2025.geolabs.fr/ogc-api returns 400 OWS error (missing service, request perameters), adding the trailing / returns 503 instead
2 GL CR /conformance JSON OK, but missing http://www.opengis.net/spec/ogcapi-processes-1/2.0/conf/core
2 EC CR Missing http://www.opengis.net/spec/ogcapi-processes-1/2.0/conf/core (note: v2)
3 EC CR Missing ../2.0/conf/oas, ../2.0/conf/oas30 or ../2.0/conf/oas31
3 CW CR OAS validation bypassed for now
4 CW CR No http://www.opengis.net/def/profile/OGC/0/ogc-process-description profile
6 CW CR Sample proceses only support jobControlOptions:[async-execute], cannot test sync case
8 CW CR Impossible to request results.yaml structure for single-output process
5 CR CW Not an error but I noticed that getting a process list uses a Link header to indicate the profile but getting the description of a specific process uses Content-Profile header. ✅ Applied & deployed (crim-ca/weaver#870 (07672b0f))
6 CR CR OGC API - Processes working, debugging server proxy error
8 CR CR Edge cases with/without outputs vs results.yaml structure with/without results profile for single-output process
8d CR CR Known case. /results/{outputID} not yet deployed.

OGC API - Processes - Part 2: Deploy, Replace, Undeploy

Test CubeWerx (CW) CRIM (CR) Ecere(EC) GeoLabs (GL)
1. Conformance Page ✅(CR) ❔(EC) ❔(GL) ✅(CR) ❔(CW) ❔(EC) ❔(GL) ❔(CW) 🚫(CR) ❔(GL) ❔(CW) ✅(CR) ❔(EC)
2. Deploy a Process ❔(CR) ❔(EC) ❔(GL) ❔(CR) ❔(CW) ❔(EC) ❔(GL) ❔(CW) 🚫(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
3. Retrieve AppPkg ❔(CR) ❔(EC) ❔(GL) ❔(CR) ❔(CW) ❔(EC) ❔(GL) ❔(CW) 🚫(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
4. Replace a Process ❔(CR) ❔(EC) ❔(GL) ❔(CR) ❔(CW) ❔(EC) ❔(GL) ❔(CW) 🚫(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)
5. Delete a Process ❔(CR) ❔(EC) ❔(GL) ❔(CR) ❔(CW) ❔(EC) ❔(GL) ❔(CW) 🚫(CR) ❔(GL) ❔(CW) ❔(CR) ❔(EC)

Notes

Test Tested Tester Note

Test implementations

Test outline

OGC API - Processes - Part 1: Core

1. Landing page

2. Conformance page

3. API definition

4. Process list

  • a) GET /processes
  • b) Verify that the response conforms to the process list schema
  • c) Verify that the overall response includes rel="self" and (optionally) rel="alternate" links.
  • d) Verify that each process summary also includes a rel="self" link and (optionally) rel="alternate" links.

    Note: the rel="self" link should be a rel="canonical" link ... oh well!

  • e) Try various limit parameter values and verify that the server supports proper paging to the process list (e.g. contains the proper next and prev links).

5. Process description

  • a) Get /processes/{processID} for one or more of the processes listed in (4).
  • b) Verify that the response is a process description.
  • c) If the server supports the OGC API Process Description conformance class, Verify that the response include a profile link header (rel=http://www.opengis.net/def/profile/OGC/0/ogc-process-description).

6. Process execution

  • a) POST an execute request to "/processes/{processID}/execution"
  • b) The request body should contain an execute request that conforms to execute.yaml.
  • c) No Prefer header
    • If the process only support async-execute verify that the server responds asynchronously (i.e. returns statusInfo.yaml)
    • Otherwise the server should respond synchronously
  • d) With Prefer header (respond-async)
    • If the process supports async-execute verify that the server responds asynchronously (i.e. returns statusInfo.yaml)
    • If the process ONLY supports sync-execute verify that the server responds synchronously
    • If the process supports either, verify that the server responds asynchronously (i.e. returns systusInfo.yaml).
    • If the wait preference is specified the server should take that into account .... although I am not quite sure how we would test that
  • e) Binary input: ???
  • f) Bbox input: ???
  • g) Value references: ???
  • h) Omit process outputs
    • a) Execute a process
    • b) Do not specify the "outputs" parameter.
    • c) Verify that the process' response contains all defined outputs.
  • i) Empty output
    • a) Execute a process
    • b) Specify the "outputs" parameter but leave it empty
    • c) Verify that the process' response is empty.
  • j) Sync execute ONE
    • a) Sync execute and request 1 output
    • b) Verify that the server responds with the response
  • k) Sync execute MANY
    • a) Sync execute and request multiple outputs
    • b) Verify that the server response with results.yaml
  • l) Preference return
    • a) Re-execute (13) setting the "return" preference to "minimal".
    • b) Verify that "large" outputs are referenced
    • c) Re-execute (13) setting the "return" preference to "representation".
    • d) Verify that "large" outputs are encoded in-line.
  • m) Job creation on sync execute: ???
  • n) Async execution response
    • a) Execute a process asynchronously
    • b) Verify that the server response with a 201
    • c) Verify that the response include a Location header
    • d) Verify that the body is statusInfo.yaml
  • o) Requesting outputs (async)
    • a) Execute a process asynchronously
    • b) When the execution has completed ...
      • verify that all requested outputs can be retrieved
      • verify that you get a 404 for outputs not requested
  • p) Empty outputs (async)
    • a) Execute a process asynchronously
    • b) Specify the "outputs" parameter but make it empty.
    • c) When process execution is complete ...
      • verify that you get a 404 for any requested output
  • q) Omitted outputs (async)
    • a) Execute a process asynchronously
    • b) Omit the "outputs" parameter from the execute request.
    • c) When process execution is complete ...
      • verify that all defined outputs can be retrieved

7. Job Status

  • a) Execute a process asynchronously
  • b) Verify that GET is supported on /jobs/{jobID}
  • c) Verify a 200 response
  • d) Verify that the response conforms to statusInfo.yaml

8. Job results

  • a) Execute a process asynchronously
  • b) Verify that GET is supported on /jobs/{jobID}/results
    • Verify that the response conforms to results.yaml
  • c) Use the "outputs" query parameter to specify the outputs that should be included in the response.
  • d) Verify that GET is supported on the /jobs/{jobID}/results/{outputID}
    • Verify that the response is returned directly

OGC API - Processes - Part 2: Deploy, Replace, Undeploy

1. Conformance page

2. Deploy a process

  • a) Compose an application package using the "OGC Application Package" or "CWL"
  • b) POST the application package to the /processes endpoint setting the Content-Type header appropriately (application/ogcapppkg+json for an OGC Application Package, application/cwl+json for CWL)
  • c) Verify that the server responds with a 200 and that the response includes a Location header.
  • d) GET a description of the deployed process from the /processes/{processID} endpoint.
  • e) Verify the process description relative to the application package.

3. Retrieve AppPkg

Retrieve the Application Package of a deployed process.

  • a) GET /processes to identify a mutable processes deployed on the server
  • b) GET /processes/{processId}/package to retrieve the application package used to create the process.
  • c) Verify /processes/{processId}/package that the server responds with HTTP code 200 and that the Content-Type is set correctly for the package (i.e. OGC Application Package or CWL).
  • d) Verify that the description of the process matches the description in the application package.

4. Replace a process

  • a) GET /processes to identify a mutable processes deployed on the server
  • b) Compose an application package using the "OGC Application Package" or "CWL" that updates the existing definition of the process.
  • c) PUT the application package to the /processes/{processID} endpoint setting the Content-Type header appropriately (application/ogcapppkg+json for an OGC Application Package, application/cwl+json for CWL)
  • d) Verify that the server responds with a 204 (or 202).
  • e) GET a description of the revised process from the /processes/{processID} endpoint.
  • e) Verify the process description relative to the revised application package.

5. Delete a process

  • a) GET /processes to identify a mutable processes deployed on the server
  • b) DELETE /processes/{processID} to delete the identified process.
  • c) Verify that the server responds with a 204.
  • d) GET /processes/{processID} to try and retrieve the deleted process.
  • e) Verify that the servier responds with a 404.