Climsoft Developers Workshop,Kitale, 18th to 22nd March 2019 - climsoft/Climsoft GitHub Wiki

Day one summary

The workshop kicked off at nine o’clock with the introduction of the participants which included participants from AMI, IDEMS, KMD,Climsoft team and Manor house (Host organisation). We then had a discussion on the objectives and the outcomes of the workshop. During this session Marcellin mentioned that the work is funded by UK Met Office.

It was suggested that, it would be great if Manor House (the host organisation) could have a weather station and join the network of volunteer stations for KMD. Samuel, mentioned that this meeting had been long overdue and its great that we were all together to discuss the challenges and how to get ahead with development of Climsoft. He mentioned also it was important to have a user centred system that meets user needs and also meets the WMO standards/specification. Marcellin then made a presentation on background of Climsoft. This included the history of Climsoft,current Climsoft governance structure, current progress in development, the unique features of Climsoft and also other existing Climate Data Management Systems(CDMs).

During the presentation there was discussions on the need for Climsfot by African Countries National Meteorological and Hydrological Services(NMHSs). This indicated that due to the following features/reasons about Climsoft, a number of Countries would like to install Climsoft version 4.

  1. It is free and open source and sustainable,hence very economical;
  2. It uses simple tools that the NMHSs don’t need to do an overhaul of what they currently have;
  3. It’s also scalable since it can be installed in observatory station.

There was also a small discussion on how R-Instat and Climsoft link up together to complement each other. We then had Samuel Machua’s presentation on the latest Climsoft version 4 installations. He outlined the Countries that had it installed which includes:

  1. Rwanda
  2. Kenya
  3. Lesotho
  4. Uganda
  5. St Vincent and Grenadines and
  6. Seychelles

During his presentation he outlined the successes and challenges encountered in each Country. He also made it clear that upgrading climsoft from v3.x to v4.x was now made easy since a small set up could be used to upgrade the interface of Climsoft version 4 and this can be shared via email or drop box to users for regular upgrades.

Out of his presentation, there was a task :- Changing forms on real time due to needs for different regions, the new form for Caribbean and Kenya agromet. A way to assign urgent tasks by Samuel to developers would be useful to get these kind of new forms done quickly.

Presentation 3

Samuel Kamau then had a presentation on Data operations at KMD using Climsoft. He mentioned that Version 4 was not fully operational here, because they are still using v3.x for data entry and verification. He also highlighted the types of products they produce currently using Climsoft and also some desired products.e.g: Gridded data.

Out of this presentation there was a discussion on how to manage units in Climsoft, a comment that even erroneous values in key entry form should be allowed to be saved (non critical errors) but they should be flagged. In addition, it was desired that limits and inter-element comparison checks be done easily for better data QC. He also highlighted the challenges they have faced in KMD in using Climsoft. One of his wishes was that from the entry forms he could have an easy way of looking into gaps in data.

Lunch at 13:30

After lunch,we had a demonstration where Danny discussed the process of using GitHub and its importance to Climsoft project. Challenges – At the beginning, there is lots of work that are put in to make things work well. He also demonstrated how GitHub is used in R-Instat and the kind of processes that any piece of work goes through before its merged. Mentioned the issues, the importance of using labels properly and milestone to organise work that needs to be accomplished by someone at certain time.

Maxwell then demonstrated the current work done on Climsoft by AMI team on data entry forms. The new metadata forms were to be demonstrated later as the dev branch was under development and had a bug that had not been fixed. Finally, we had a round table discussion with the AMI team on the challenges and their experience on developing Climsoft.
Among the challenges that were raised included:

  1. There was no enough data in the database to test the functionality of forms (It was agreed that there could be open data that could be shared, possibly generated data or some closed development data that developers have to consent that they won’t distribute);
  2. The database model was not made available to us in the beginning and so it was tough to get around things-Some of the decisions behind this;
  3. There was no availability of a specification document – we now have WMO CDMS specifications document;
  4. Communication issues; and
  5. Networked test environment.

Maxwell mentioned that his experience in Rwanda seeing the software being operational was an eye opener. The tasks that came up from the discussions from the first day were:

  1. Customised data entry forms;
  2. Users and administrators to be able to identify gaps;
  3. Discussion on how to manage limits;
  4. Gridded products;
  5. Making it easier to complete metadata;
  6. Permission based interface;
  7. Settings – make software more flexible;
  8. Replacing grid in the form selection;
  9. Discussion on merging of development branch;
  10. Data for developer testing;
  11. Tutorial database;
  12. Looking at the data model when we are all together; and
  13. Setting up a network testing environment.

Day two Summary

The previous report of the meeting was shared on GitHub so that Climsoft developers and those interested can check out for any updates.
Ian gave a presentation focusing on the GitHub project board feature, which gives view of what is happening with every developer on a project.The feature allows the project contributors to give themselves timelines on a task and view its progress until its ready for review and complete.

We split into two groups. One group discussing on the future Climsoft Data model and the other had to discuss on possible ways for accommodating Portuguese and French translation. The team that was to make designer changes on dialogs did the following:

  1. On all forms,except metadata form which had a bug, labels were placed on top of combo boxes to prevent them from overlapping into the combo boxes;
  2. Some user controls had to be newly aligned to give room for the lengthy English to Portuguese/French translated labels; However, there were some misalignment in English labels to attain alignment for Portuguese/French translated labels. It was suggested that for check boxes some long translations could be split into two lines. On the other hand,the team to improve on Climsoft Data model agreed that:
  3. No significant changes are to be made now on Climsoft version 4 so that user don’t need to change the data model;
  4. From version 4 to version 5,there would be changes in the data model; 3)For quality control,history for each entry, and the operator who performed the change will be recorded for future reference;
  5. Later Climsoft Versions will have a single observation table instead of observation initial and observation final;
  6. In Climsoft version 5 users will be required to import data from Climsoft version 4 and there would be repeated entry in the observation table;and
  7. In Climsoft version 5 the web application will be limited to data entry only but in Climsoft version 6, the web application will retire the desktop Cimsoft version while R-Instat will be installed on the server to generate products. Finally,it was suggested that all the raised issues were to be put on GitHub and some time set aside so that the team can work on the data entry forms before the end of the workshop.

Day three summary

First Session by Samuel on WMO CDMS Specifications

After the recap, Samuel Machua presented on Climate Data Management System Specifications. He explains how CDMS works:

  • Under this he mentioned on the Implementation of CDM’S to serve the users
  • That WMO came up with CDMS specifications to suit the users

Contributors

The three major contributors were main contributors, other contributors and reviewers. From each of these, examples were provided for each case. Under the contributors,he mentioned that the specifications come from the people who have participated. Major Components of These components were displayed in a graphic form and they includes;

  • Data presentation
  • Data delivery
  • Data management
  • CDMS governance
  • Data analysis All these components originate from climatic data. For all these components to work, there must be an IT structure.

Climate Data

This represents a wide range of time series climate data. It includes the following:

  • Global climate Observing System(GCOS)
  • Metadata on observation
  • Standard WMO products
  • Derived observations and gridded data
  • Output from the numerical models
  • Spatial and impact data.

CDMS governance The CDMS governance components refer to a consistent set of policies and governance processes needed to build a solid foundation for the establishment and management of climatic data and related services. These components includes; data policy and governance policy. Examples were given for each of them.

Data Management Addresses the functionality required to manage climate data effectively and it includes;

  • Data ingest and extraction
  • Data rescue
  • Observations quality control
  • Quality assessment
  • Management of climate metadata

Data delivery Refers to the functionality required to deliver climate data and includes; data discovery for both climate and climate metadata, data delivery in WMO formats and data delivery based on open spatial standards.

Data analysis This involves a wide variety of analytical techniques that are applied to climate data, for example; a series of techniques, including statistical, spatial and image analysis, homogenisation, numerical modelling analysis.

Data presentation Represent a diverse set of techniques used to communicate climate _ related information. These include;

  • Written reports
  • Time series climate data exploration via a graphical user interface with functionality such as; generating a broad variety of business intelligence reports, including tables, graphs, scatter plot, histograms and ensembles.
  • Multimedia exploration of data via, for example, podcast, videos or photographs.
  • Allowing the download of viewed data via the graphical user interface.

IT Infrastructure It represents the functionalities required to support support a CDMS. After going through the major components of the CDMS, We looked at the over view of CDMS functional components. Examples on how Climsoft meet the CDMS specifications were demonstrated. This was done using form synoptic where data is transmitted after every 3 hours.

Component Classification scheme This scheme is used in detailed exploration of CDMS components to indicate whether functionality is required, recommended, or optional. These three are identified using three different colours namely; Yellow for required, Green for recommended and Gray for optional. This classification can be used to assist NMHSs in determining what functionality they are able to implement.

WMO standard products This includes; routine messages, climatology standard normals, CLIMAT, World weather records and Aeronautical.

Data Ingestion This includes; WMO messages, Vector, Roster array, other formats, Status log. We then looked at some of the skills required in the implementation of a CDMS.

Second Session by Ian We discussed on how to remove multilingual app tool kit and creating our own resource file containing all the translations. The decision would be made on which column will be in the resource file. This issue was added as #200 on gitHub.

Third Session In this session, we divided ourselves into two groups whereby the first group was working on key entry forms i.e moving the controls from the forms to the user control. While the second group was working on adding the Climsoft related issues on github. We also had an optional farm tour to one of the Manor house founders later in the evening.

Day Four Summary

Morning Session A recap of the previous day (Wednesday) activities was made by Wycliffe (AMI). Ian then led a session on Climsoft Installation process. He initiated the discussion with reference to two issues related to the Climsoft installation i.e. issue #229 (manual steps required when installing Climsoft) and #404 (Climsoft installation procedures).

Ian demonstrated step by step procedure on installing Climsoft. The following were discussed and agreed upon:

  • It was noted that generally most users do not read instructions on the ‘read me file’ and the license agreement.
  • Further discussion addressing when and why it is necessary to install Climsoft as an Administrator would be made.
  • A new advanced installer (refer to #479 on Github) would be created and it will have the following effects:
  • The Microsoft Office runtime and Visual Studio runtime will no longer be necessary.
  • The advanced installer would automatically detect whether .net framework is already installed (if not, launch the installation).
  • It was agreed that the WR Plot software be removed (not bundled with the installer). It was recommended that Users get their own WR Plot software licenses.
  • MariaDB would be removed from the installer. It was agreed that there will be two set up options i.e. Climsoft set up and MariaDB set up on the github release page. The user will have an option to click on either options depending on their requirements (no need to install MariaDB if they already have one). When the user attempts to log in to Climsoft for the first time, a welcome screen will pop up with two options: connect to a database or create a database. If they do not have MariaDB installed, a set of instructions will appear guiding them through the installation. Installation instructions on the text file (read me file) will be removed from the launcher and placed on the release page on github. Team then split up into two: One group documented issues arising on github and the other group (mainly composed of the AMI team) carried on with work on the forms.

Afternoon session Samuel and Patrick led a discussion on the metadata forms. Patrick explained the rationale behind redoing the metadata forms using custom controls. The custom controls have inbuilt validation and inbuilt connectivity to the database.

  • The following issues were agreed upon during the afternoon session:
  • Priority would be given to work on the data entry forms as this is important for the next release of the software.
  • The software will be set in such a way that it allows users to input a different data type other than the data type specified in the database but with a warning message.
  • The AMI team would look through all the metadata forms and check if there are fields or tables that should have different data types other than those specified in the database (i.e. assess if the data types are appropriate). The team should then raise such issues on Github.
  • Designer changes will be made on all metadata forms by moving the Cancel, Close and Help buttons to the base.
  • Use ISO 86 date strings on the Opening and Closing datetime fields on the Station metadata form.
  • Everyone would take time to think through ways to update key controls on the data entry forms efficiently and effectively. This would then prompt further discussions.

DAY 5 SUMMARIES

We received recap from the previous day from Yvonne. After this, a discussion began around the issue of editing the key controls on the Data Entry forms. A number of potential solutions were raised including leaving the forms to work as they are and come up with a way of making edits at data entry level easier without introducing complexity or potential errors. An addition to this was to have a background color change on key controls until values are entered into the Value Flag Period controls. Implementation of this is to be done and reviewed. Samuel also raised a few issues the AMI team should be keen on when creating the Agromet forms using the custom controls. He emphasized the tab order should be vertical. He also had other specifications to the Caribbean form which he will share with the team later on in the day.

CLIMSOFT ROAD MAP Ian took the team through the Climsoft 5 Year Road Map The Road Map was reviewed and various changes were made to it. A number of issues stood out during the discussion with some of them being:

  1. The terms to be used in the Road Map during the meeting in May. This was to ensure that everything in the Road Map is in line with the WMO Open CDMS Specifications and also to properly communicate the plan.
  2. Strengths of having multiple databases being supported by the software were highlighted
  3. Other discussions were around clarification on the Climsoft App and what it would entail. Ian then gave a brief description on the various stages of the Road Map with regard to the versions. Ian will avail the revised Road Map after making the proposed changes (in the following week). Ian then shared with the team what the May meeting will entail. This requires clarification from Steve.

DISCUSSION ON THE NEW DATA ENTRY FORMS Samuel shared the script for the latest data model with the team. He took the AMI team through the specifications for the Caribbean and Agromet forms; Issues that arose;

  1. Validation of the controls (how and when it should be done)
  2. Minimizing Pop-ups
  3. Functionality of Clear and addition of the Cancel button.
  4. Flags should be read only and heading included.
  5. Agromet Forms specifications.

Project Board and Milestones Ian led the team on going through the GitHub Project Board and Milestones He created a Project called March Sprint where a few issues were added. The AMI team was tasked with creating issues on GitHub relating to the respective tasks they have. It was agreed that specifications on tasks should be made to effectively assist in implementing the tasks. The workshop was concluded with every member of the team and the Manor House representatives sharing their experiences of the workshop.

Workshop Participant

Flow diagram showing process for making contributions