Climsoft CDMS Architecture and Implementation - climsoft/Climsoft GitHub Wiki

This document is currently a incomplete draft

Overview

Climsoft was originally implemented using the Microsoft Access database and evolved to use Visual Basic and MySQL. Version 4 was rewritten in Visual Basic .NET and released as open source software under a GPL v3 License.

This document outlines implementational details for a future version of the software called Climsoft CDMS.

We will refer to previous versions of Climsoft, that were limited to Microsoft Window, as Climsoft Desktop for Windows.

In 2019-2020, an additional product called Climsoft Web Entry will be created to enable data entry using any device with a web browser.

The Climsoft 5 year roadmap gives details of the time frame over which we will retire Climsoft Desktop for Windows and enhance Climsoft Web Entry to create a multi-device/multi-platform solution called Climsoft CDMS.

Climsoft CDMS is intended to have a single code base (simplifying development) whilst providing web, desktop and mobile apps on multiple platforms including:

  • Desktop solutions for Windows, Linux and MacOS
  • Mobile app versions for iOS and Android
  • A fully functional version accessible over the web that can also be used offline when a connection to the internet is not available.

Climsoft CDMS aims to implement all of the required sections of the WMO Climate Data Management System Specifications and become a WMO-certified CDMS.

Principles

The software architecture for Climsoft CDMS is guided by the following principles:

  1. Compliance: Conforms to the WMO Climate Data Management System Specifications
  2. Open source: Continue to build and utilise open source software
  3. Open data: Encourage progress towards 5* open data
  4. API-first: Prioritise development of the Climsoft CDMS server software, including the API and business logic, first. Initially implementing a prototype client for Climsoft Web Entry
  5. Don't repeat yourself (DRY): Single solution/code base providing solutions for desktop, web and mobile without duplicating effort by maintaining separate code bases for different platforms
  6. Offline-first: Ensure that tools can continue to be used offline when internet connections are not available and synchronise data when a connection is available
  7. Microservices: Microservices should be used where appropriate instead of creating a single monolithic server-side architecture

Climsoft CDMS Server

Climsoft CDMS Server can be installed nationally, on a physical machine in a server room, or in the cloud (or a combination of both).

Each server installation can include one or more componets designed to expose the CDMS API, the web application and web services as microservices. All external connections are served over secure HTTPS.

Climsoft CDMS Server architecture

CDMS API component

Central to Climsoft CDMS Server is the core CDMS API component. All request for data will be made through Web API calls. Since all data requests pass through the API layer, the CDMS API componet will be responsible for implementing and enforcing all of the business logic that is required. This means that core business logic will not be implemented in clients or enforced solely by the database.

Web App component

The Web Application component is responsible for delivering a Single Page Application (SPA) Progressive Web App (PWA). Unlike classical web applications, the progressive application will have the following characteristics:

  • Progressive — Work for every user, regardless of browser choice
  • Responsive — Fit any form factor: desktop, mobile or tablet
  • Connectivity independent — Work offline, or on low quality networks.
  • App-like — Feel like an app to the user with app-style interactions and navigation
  • Fresh — Always up-to-date due to Over The Air (OTA) updates
  • Safe — Served via HTTPS to prevent snooping and ensure content hasn't been tampered with
  • Discoverable — Identifiable as an “application” thanks to W3C manifests allowing search engines to find it
  • Re-engageable — Make re-engagement easy through features like push notifications
  • Installable — Allow users to keep apps they find most useful on their home screen without the hassle of an app store
  • Linkable — Easily shared via a URL and does not require complex installation

Web Services component

The Web Services component will be incorporated into Climsoft CDMS server using existing modules that have been created for GeoNode (see Climate Data Framework below). This can include:

OGC Service Acronym Specification section Classification Notes
Catalogue Service CSW - -
Web Mapping Service WMS 8.1.1.1 Recommended
Web Feature Service WFS 8.1.1.2 Recommended
Web Coverage Service WCS 8.1.1.3 / 8.1.1.5 Recommended Including NetCDF

Technologies

Operating Systems

Following principle 2, Climsoft CDMS recommends the Linux operating system over Microsoft Windows for the main Climsoft CDMS server. To simplify support and maintainance we will officially support a single version of Linux, but users may choose to deploy the software in different environments.

Ubuntu

The Ubuntu Linux operating system software has been chosen because:

  • It is already in use by some Climsoft users
  • Ubuntu is the defacto OS for geospatial software on Linux (e.g. adopted for OSGeo Live which is reference in the WMO CDMS specifications)
  • Building software for alternative operating systems, like Enterprise Linux (e.g. CentOS), is more difficult

Databases

PostgreSQL (PostGIS)

The PostgreSQL database, with PostGIS spatial extension, is recommended for the Climsoft CDMS database server. PostGIS is the most mature open source spatial database and, at the time of writing, is the only option with support for raster fields (required to support gridded data).

MySQL / MariaDB

Climsoft CDMS will continue to support MySQL/MariaDB, however it is anticipated that some of the more advanced capabilities may not be available if these databases are chosen due to MySQL not conforming to the OGC standard

Web server

NGINX

Most documentation for the modern frameworks now focuses on NGINX. Apache2 is still used for traditional Linux, Apache, MySQL and PHP (LAMP) implementations, but has mostly been superseded for other applications.

Programing Languages

A number of options have been considered for the primary server-side programing language including PHP (used by CliDE CDMS), Ruby of Rails, NodeJS and Python.

Python

Python 3 has been chosen as the primary server-side language due to it's strength in delivering robust web back-ends whilst also having an extremely rich set of scientific and geospatial libraries. Python is equally suited to new programmers who require a simple way to get involved as it is to experience developers who require powerful capabilities and performance.

R

It is anticipated that other languages will be support on the Climsoft CDMS server. A key example is the R programming language which will be used alongside Python to create a set of existing products that are required.

Frameworks

Python web frameworks include:

  • Microframeworks (e.g. Flask)
  • Full stack framework (e.g. Django)
  • Scientific/analytic frameworks (e.g. Dash)

Microframeworks are a good choice for simple sites that are possible only required to expose an API. As the software becomes more extensive it is likely that many of the capabilities of a full stack framework would have to be recreated.

At least one previous attempt to quickly create a prototype of a web front end for Climsoft used the Shiny scientific framework from RStudio. R and Shiny have not chosen as the primary language and framework for Climsoft CDMS due to Shiny having a dual licence and concerns about deploying the software securely in production without relying on the ShinyApps proprietary cloud solution, or having to modify Shiny Server for enterprise use without users having to purchase licences for Shiny Server Pro.

A comparable scientific web framework in Python, that also has dual licencing, is Dash) created by Plotly.

Django Framework

Django (GeoDjango) became the clear choice of python framework due to its broad set of capabilities, prioritisation of security, potential for rapid development and overall flexibility. Although the Django framework is a suitable choice for delivering an API, its real strength will show when additional scientific and and geospatial capabilities are required.

Another advantage of adopting Django is that Met Services may find if relatively simmple to create a secondary web server that can host additional services for users that consume Climsoft data. These services can be tailored to different user groups and provide bespoke tools and information.

Application Programming Interface (API)

The intention of the Climsoft CDMS API is to provide secure and controlled access to the Climsoft database via a single interface. CDMSs have complex business logic and it is important that this is not enforced solely by the database or implemented only in a single client.

Django Rest Framework

Django Rest Framework has been chosen over the alternatives due to it being the mainstream solution and offering self-describing APIs.

Climate Data Framework (SDI)

Section 11.1 of the The WMO Climate Data Management System Specifications discusses the need for a future Climate Data Framework spatial data infrastructure (SDI).

By using Django as our primary framework we also may benefit from Django modules that were developed for GeoNode. GeoNode is built with Django, shares similar dependencies and has a GPL v3 licence. See dependencies (e.g. currently Python 2.7), core Django modules and requirements*.txt.

Climsoft CDMS potentially fits the GeoNode 3 architure vision, and would be a good fit as a "Downstream project - Django project with PostGIS backend". GeoNode 3 does not currently exist but, given the existing parallels, there may be benefits to aligning our architecture further.

GeoNode vision architecture

Due to the similarities, we can learn from decisions made in the current GeoNode implementation and the discussions and vision for the next major version.

Also note: The Django project uses Transifex for translation

Summary of server-side technologies

Currently, the anticipated main components of the server-side solution are:

Provision and deployment can be handled by Ansible. It is anticipated that much of the work to set up production environments using the above libraries will be simplified by using ansible-django-stack.

Climsoft CDMS Client

This section on the Climsoft CDMS Client has not yet been completed

The aim of the Climsoft CDMS client is to develop and maintain a single solution/code base whilst eventually being able to provide multi-platform solutions for desktop, web and mobile.

The main principle of the client is to ensure that users can still use the software offline when they have limited internet connectivity.

Initially a lightweight prototype web client will be created that does not implement all of the features of a progressive web application, but can be deployed to early adopters and also used for testing API and business logic (principle #3). Longer term, a modern JavaScript framework will be used to create a single page application progressive web application that can be deployed to multiple platforms.

Developing for offline

The components of an offline-first application are described well in the Offline Web Applications Udacity course.

JavaScript frameworks

Essential reading:

Both Angular and ReactJS can be used to develop progressive web apps that use service workers to allow offline use.

Angular ReactJS
Created by Google Facebook
Core size Large MVC framework "batteries included" Smaller core, focused on delivering Views
Type Framework (inversion of control) Library
Language Typescript JavaScript (/TypeScript)
Paradigm Object Oriented Programming (OOP) Functional Programming (FP), portable/testable
Approach Highly opinionated, one way to do something Unopinionated, many approaches
Barriers Steep initial learning curve, but becomes easier due to everything included Easier initial adoption, but becomes harder due to need to research the best way to implement something
Upgrades Potentially easier to upgrade because all components of the framework get updated together Harder to upgrade when third-party modules are not updated in time with the main ReactJS releases
Hybrid development Cordova / Ionic Cordova / Ionic (v4+ or third-party)
Native deployment Third-party NativeScript React Native

Current main considerations:

Advantages Disadvantages
Angular Compelling due to provision of an entire ecosystem Do we need everything provided? E.g. Our Models will be developed and managed in Django, the JavaScript framework is only really needed to consume the CDMS API
All components are updated together with release updates Steep learning curve, possibly focusing on components that we will not require much use of
ReactJS The smaller core contains the main components we need Risk of a steep learning curve later in the project when possible solutions need to be researched
Parallel projects like GeoNode are converting client-side code to ReactJS Concerns that contributed modules may not be updated inline with ReactJS releases or may become unsupported

Other lighter-weight JavaScript frameworks have also been considered from Vue.js to jQuery Mobile. Among the smaller JS frameworks Vue.js certainly shows a lot of potential, especially the soon-to-be-released Vue.js 2.0. Vue has currently been ruled out for this risk-adverse project due to it's lower overall market share, but this situation may change in the future.

Targeting multi-platform

Targeting multi-platform gives key advantages:

  • Users of the App will have access to additional capabilities, like direct access to the device's camera to upload photos to metadata, plus receiving reminders and push notifications

Apache Cordova

Cordova allows us to develop a single codebase that can be deployed on desktop, mobile and web. This will allow the creation of desktop and mobile apps instead of relying only on CSS and HTML templates to produce mobile websites. The user therefore won't be forced to use the application in a mobile browser app like Chrome or Safari, but rather an actual (hybrid) app is created using a low-level "browser shell" (like iOS’s UIWebView or Android’s WebView).

Ionic framework

Ionic is a front-end UI framework that builds on Apache Cordova and the JavaScript framework with additional components to improve the look and feel and UI interactions.

From version 4 onwards, the Ionic framework can be used with both Angular and ReactJS.