News - dlr-eoc/prosEO GitHub Wiki

News

2025-01-29: prosEO version 1.1.0 released

The new prosEO release 1.1.0 brings a lot of small improvements in functionality and many bug fixes, but the main change is under the hood with an update of the base libraries to Spring Boot 2.7.18 (from 2.1.6).

Changes relative to version 1.0.1 are:

New functionality

  • GUI:
    • Processing Order list now remembers sort, page and selection properties after actions
    • Display function for Product Archives added
  • Production Planner:
    • Product queries of orders with deferred execution time are re-evaluated before start (issue #229)
  • Storage Manager:
    • Now forcing path style access to objects when using the AWS V2 SDK client to support simpler object storage implementations
    • Backend storage re-organized based on upload date and time
  • AUXIP Monitor:
    • Lenient parsing of time stamps to handle product metadata not compliant with AIP/PRIP/AUXIP ICDs
  • AIP Client:
    • Lenient parsing of time stamps to handle product metadata not compliant with AIP/PRIP/AUXIP ICDs
    • Product attribute completeness check relaxed
    • Extendable JAR and extension hooks for mission-specific extensions added (and mission-specific code for Sentinel-1 removed)
    • Ignore (sometimes scrambled) sensing stop times in product metadata from archive when checking local files for duplicates
  • Order Manager and Production Planner:
    • Log Processing Order history
  • ODIP:
    • Production metrics added

Library updates

  • Spring Boot dependency updated from version 2.1.6 to 2.7.18 including dependency updates for Hibernate and other libraries
  • Replaced RestEasy by Apache HTTP Client 5 in BaseWrapper

Data model changes

  • Processing Order History added
  • API Metrics for collection of production metrics added

Performance

  • Obsolete use of semaphores between Ingestor and Production Planner removed (issue #208)
  • AIP Client:
    • Skip order generation and wait if product is online
  • GUI:
    • Use Order Manager instead of Production Planner to delete orders

Bug fixes

  • GUI:
    • Several fixes in Processing Order detail view
    • Disabled processing facilities not shown in order planning menu (issues #196, #218)
    • Dashboard also shows closed job steps as completed
    • Processing Order deletion fixed
    • Fixed some cases where lookup of objects by database ID would not work
  • AUXIP Monitor:
    • Basic authentication fixed
  • CLI:
    • Fixed handling of Workflow attributes (issue #202)
  • Logging:
    • Several typos corrected and messages clarified
    • Banner added to Monitor log
  • Deployment:
    • application.yml files updated
    • Test orders in single-node deployment corrected
  • ODIP:
    • Input product selection by product class and time range fixed
  • Production Planner:
    • Transaction handling fixed in several places
  • Storage Manager:
    • Handling of parallel access to cache files fixed

Documentation

  • Data model description updated (new document version 1.7)
  • Java and REST API reference updated
  • Clarifications and corrections in CLI syntax documentation/help text (issues #219, #226)
  • Java API docs for Processing Order History added

2024-03-18: A new version (1.0.1) of prosEO has been released

This release includes changes to the Storage Manager not yet tested in a full load scenario, therefore it is considered intermediary and may not be fully ready for production use. A full load test is scheduled for late April/early May 2024.

This version includes the following changes (compared to the previous version 1.0.0):

  • New functionality:

    • No changes

  • Data model changes:

    • No changes

  • Security fixes:

    • No changes

  • Performance:

    • Storage Manager: Improved cache handling for uploads and downloads to external clients

  • Minor fixes:

    • ODIP: Retrieval of workflows without workflow options fixed (Github issue #203)
    • Storage Manager: Concurrency issue on cached files fixed
    • Ingestor, Production Planner: Obsolete code for semaphore handling removed
    • All services: Superfluous property "module.version" removed from all pom.xml's
    • Product Planner: Unit tests added

  • Documentation:

    • No changes

No new CLI client has been provided, the client for prosEO 1.0.0 is still current.

2024-03-07: We finally made it to prosEO 1.0.0, the first official release!

This version includes the following changes (compared to the previous version 0.9.5):

  • New functionality:

    • Production Planner: The ordering strategy for job steps is configurable (either by sensing start time or by order submission time, Github issue #210)
    • AIP Client: The number of parallel product download orders is now configurable

  • Data model changes:

    • No changes

  • Security fixes:

    • No changes

  • Performance:

    • No changes

  • Minor fixes:

    • ODIP: UUID now mandatory for order creation (Github issues #205, #206)
    • ODIP: When retrieving workflows the "enabled" flag is now respected (Github issue #209)
    • Ingestor: Ingestion now works for existing products without product file
    • Ingestor: Transaction handling for Planner notifications fixed
    • GUI: URLs for core services are now properly escaped (Github issue #159)
    • Logging: Line breaks in prosEO messages now replaced by blanks (Github issue #200)
    • Order Manager: Deletion of a mission now works even if spacecrafts and orbits exist
    • AIP Client: Failure notification through Notification service fixed
    • AIP Client: "Download all by time" method fixed (now checks delta between locally and remotely available files and downloads missing files)
    • AIP Client: Now leaves generation of product UUIDs to Ingestor to avoid duplicates
    • Several log messages corrected

  • Documentation:

    • prosEO Wiki in Github completed
    • Several improvements/corrections in internal documentation and README files
    • Test data generation updated
    • Copyright notice in GUI updated for 2024

A new CLI client has also been provided, which should be used with all prosEO instances using prosEO 0.9.5 or prosEO 1.0.0. It can be found at the prosEO artifact repository.

2023-12-12: A new version (0.9.5) of prosEO has been released.

This version includes the following changes:

  • New functionality:

    • ODIP Service added for On-demand processing API
    • AIP Client Service added for data retrieval from Long-term Archives
    • Archive Manager added for LTA endpoints
    • Workflow Manager added for new Workflow entity
    • Notification Service added for alerting
    • CADIP Monitor: Endpoint status check added, download efficiency improved
    • Selection policies: LargestOverlap and LargestOverlap85 implemented

  • Data model changes:

    • Product: File class optional
    • Workflow entity added
    • ProductArchive entity added

  • Security fixes:

    • ClamAV cron job updated in bastion host deployments
    • Prevent rkhunter warning for missing SSH protocol entry
    • Unsafe JPQL generation fixed in Group Manager
    • Hide passwords in facility commands in CLI

  • Performance:

    • Remove semaphores from Production Planner
    • Retries added for DB transactions failing due to isolation Repeatable Read
    • Handling of database transactions reworked

  • Minor fixes:

    • Missing error codes corrected
    • Documentation corrected and updated
    • Log and help messages improved
    • General code cleanup and in-code documentation update
    • Increase of database connection limit in PostgreSQL image
    • Selection policies: ClosestStart/StopValidity corrected
    • ProductQueryService: Unwanted query updated fixed
    • GUI: Retrieval of order state fixed
    • PRIP API: GeoFootprint as LineString for invalid polygons
    • AUXIP Monitor: Change HTTP client library
    • Storage Manager: AWS SDK dependencies updated, code of first implementation disabled

A new CLI client has also been provided, which must be used with all prosEO instances using prosEO 0.9.5.

2023-03-16: A new version (0.9.4) of prosEO has been released.

This version has the following changes:

  • GUI Usability:
    • Order of menu items in order menu changed,
    • Grafana dashboards added for monitoring (at endpoint ".../proseo/grafana", requires separate authentication for now),
  • Performance and technical improvements:
    • Database model updated (entity naming improved, Github #134),
    • Order cleanup split in separate transactions per order (Github #183),
    • HTTP connection timeout in BaseWrapper implemented (Github #153),
    • Production Planner offers choice between synchronous and asynchronous planning,
    • Separate database transactions for facility information in Production Planner,
    • Performance in order closing improved in Production Planner,
  • Functionality:
    • New implementation for Storage Manager,
    • More checks for mandatory fields and "null" values for lists in REST APIs (Github #129, #174),
    • GeoFootprint field added to PRIP API (PRIP ICD issue 1.9),
  • Bug fixes:
    • Mission deletion function corrected,
    • Publication time truncated to milliseconds in Ingestor (PDGSANOM-9794),
    • Fixed error in GUI, which led to the suppression of selection rules valid for all configured processors
    • Ingestor now skips synchronization with Production Planner, if no Planner is configured,
    • Fixed error in BaseWrapper, which led to a wrong Job Order file being handed over to the processor CFI,
    • Some messages fixed
  • Documentation:
    • Updates to deployment READMEs (not a functional change).

A new CLI client has also been provided, which must be used with all prosEO instances using prosEO 0.9.4.

2022-11-14: A new version (0.9.3) of prosEO has been released.

This version has the following changes:

  • GUI Usability:
    • Filter for "COMPLETED" jobs now also includes "CLOSED" jobs,
    • Column headers for start and stop times differentiated between sensing start/stop times (at job level) and processing start/stop times (at job step level),
    • Retrieve log files from running jobs faster by getting them directly from Kubernetes without database access,
    • Display of selection rules by mode and configured processor,
  • Performance and technical improvements:
    • Performance improvements in Production Planner by using native queries with scalar results instead of retrieving complete objects,
    • Performance improvement in Production Planner (expected) when closing processing orders,
    • Use of AWS S3 Transfer Manager (V1 API) for multi-threaded product download in Storage Manager,
    • Logging unified with centralized repository for messages,
  • Functionality:
    • Value type for product attributes corrected in PRIP API,
  • Bug fixes:
    • Synchronize product metadata updates in Ingestor with Production Planner (addresses concurrent database update issue in VDAFOP wrapper),
  • Documentation:
    • Updates to deployment READMEs (not a functional change).

A new CLI client has also been provided (unchanged w.r.t. CLI version 0.9.2 except for the logging, but should be used if on CLI version 0.9.1 or earlier).

2022-09-08: A new version (0.9.2) of prosEO has been released.

  • GUI Usability:
    • Job state selection changed to menu style,
  • Performance improvements:
    • Keep the dispatch cycle in the Production Planner from checking job steps while the finishing of job steps runs,
  • Bug fixes:
    • Bug in jump from dashboard to job step in order fixed in GUI,
    • Behaviour for order closing corrected in CLI.

2022-09-01: A new version (0.9.1) of prosEO has been released.

This version has mostly fixes for bugs and performance in the backend services and a few functional changes. The most notable are:

  • Functionality:
    • Filtering of jobs by completion state added to GUI.
    • Display current prosEO (GUI) version in the footer.
  • Usability:
    • Job steps within jobs are now planned and released in ascending order of database ID for a more predictable planning/releasing progress.
    • Job steps can be grouped in blocks for handling in the same database transaction during order planning and release.
    • When refreshing an order detail page, the currently selected page stays selected.
  • Performance improvements:
    • Speed improvements for order releasing in the Production Planner.
    • Dispatcher cycle does not run concurrently to order releasing in Production Planner.
    • Improved retry functionality when accessing an S3 backend storage from the Storage Manager.
  • Bug fixes:
    • Improved handling of semaphores in Production Planner.
    • Cache clean-up now relies on cached file sizes instead of (inaccurate) current operating system info in Storage Manager.

2022-07-27: A new version (0.9.0) of prosEO has been released.

This version has some functional enhancements and some bug fixes in the backend services. The most notable are:

  • Functionality:
    • Footprints and "IN" operator added to PRIP API.
  • Usability:
    • Jobs are now planned and released in ascending order of sensing start time for a more predictable planning/releasing progress.
    • Job step logs can now be downloaded directly to the local workstation (and are not transmitted for all job steps on each display of a processing order).
    • Messages for CFI wrappers improved.
  • Performance improvements:
    • Closing a processing order removes all links to input products, reducing the amount of links between products and the job steps that require them as input.
    • Avoid locks in Production Planner caused by concurring planning and releasing threads.
    • Some of the automatic prosEO-internal triggers are now configurable and can be switched off.
  • Bug fixes:
    • Several bug fixes in the PRIP API, which surfaced during tests for Sentinel-1B and for Sentinel-5P.
    • Showing jobs by state in the CLI now works.
    • Processing orders can now be generated for orbit sequences spanning the 9999/10000 border.
    • Boundary case error in ValIntersect/LatestValIntersect selection policies fixed.
    • Developer documentation updates: During several new installations the README files (especially for single-node deployment) were clarified and enhanced.
    • Automatically generated documentation of the CLI syntax added on Github (https://dlr-eoc.github.io/prosEO/proseo-ui/proseo-ui-cli/syntax.html).