G1. TECHNICAL OVERVIEW - colouring-cities/manual GitHub Wiki

Introduction

This section is written as an introduction for technical and non-technical CCRP academic collaborators wishing to set-up and run Colouring Cities platforms, as well and those wishing to understand set-up steps.

Key links

Colouring London/CCRP software engineering history summary 2016-2023

Technical architecture for Colouring London, GitHub repositories and open licences, were selected and set up by Tom Russell, at the Centre for Advanced Spatial Analysis (CASA), University College London between 2017 and 2019. Colouring London's back-end was initially designed to perform four main technical functions to meet the brief for a free platform able to use restricted OSMM polygons to capture and visualise building attribute data. These were as follows: storing OSMM footprints and source metadata; rendering map tiles from collected data to allow data to be visualised on the platform (in a way that did not compromise Crown copyright restrictions on OSMM geometry or footprint coordinate release); permitting crowdsourcing and public editing of building-attribute data; and facilitating access to open data downloads. Interface and content design has been developed since 2106 by Polly Hudson at UCL and The Alan Turing Institute, working in close collaboration with UK RSEs (listed below).

tom2 . Image produced by Tom Russell 2017.

In 2020 Colouring London's development moved to The Alan Turing Institute where the international Colouring Cities Research Programme was set up. This first forking of code was carried out in 2019 by the American University of Beirut, followed by collaborations with Bahrain, and Australia. Between 2021 and 2022, technical work on Colouring London CCRP core code was overseen by Turing's research engineering group and the code base for a new core colouring repository platform and for Colouring Britain roll out. During 2022 Tom Russell, now based at the University of Oxford funded international RSE meetings as part of his Software Sustainability Institute fellowship. Work also began on Colouring Indonesia. Between 2022 and 2023 the core repository was also supplemented with over 50,000 lines of code,for the 'Planning' section to support streaming and visualisation of planning data by application status at building level (part of a collaboration between the Turing and Loughborough University). Colouring Colombia and Colouring Germany also came on board in 2022, with Colouring Australia officially launched in the same year.

In January 2023 work was completed at the Turing on the new core colouring GitHub repository. This now allows for co-working on core code across countries- as well as discussions on country specific code-, and for improved synching between international platforms with work beginning work on Colouring Sweden and from the summer Colouring Canada. In October Colouring Beirut which, had temporarily paused, began to be reactivated with a larger team. The new core repository is already resulting in rapid progress with the code base being made. Pooling of knowledge and time also helps maximise efficiency and platform quality and help reduce platform engineering costs for all partners. In addition it allows for the organic development of an ever expanding, friendly, collaborative expert technical group through which engineering advice and experience of the project can freely shared between both existing CCRP partners and new engineers coming on board.

For the rest of 2023 the Alan Turing Institute work will focus on: supporting co-working on core colouring code across countries; assisting new international academic CCRP partners to come on board (One-to one technical discussion are offered by The Alan Turing Institute to all CCRP partners, with input from the University of Oxford, and are encouraged at demo set-up stage); set-up and testing of Colouring Britain using open footprints; set-up and comparison of self-funded, regional university open data coordination hubs (beginning with Colouring Loughborough and involving a comparison with the Colouring Australia distributed architecture model); joint papers and CCRP data category analysis/data capture testing initiatives. During the year discussions within the CCRP group regarding set up of global region hubs, and the use of platforms as tools for emergency data collection in disaster scenarios, will be continued. Technical tips and information considered of potential value to the CCRP technical alliance, in addition to GitHub documentation, will continue to be added here by the Turing technical team as work progresses in consultation with CCRP partners.

International CCRP Software engineering teams

CCRP technical coordination and Colouring London prototype/Colouring Britain development:

Dr Mike Simpson (University of Newcastle, 2022-) Advisory support:
Tom Russell - advisory/original technical lead-(University of Oxford, 2017-)
Mateusz Konieczny (Freelance/OpenStreetMap specialist/currently working with Loughborough University, 2020-);

(Alumnae 2018-2022: (Ed Chalstrey (Alan Turing Institute Research Engineering Group 2021-22), Dr Flora Roumpani (Alan Turing Institute, 2020), Maciej Ziarkowski (UCL, 2019-20), Dominic Humphrey (UCL, 2018), Melda Salhab (UCL, 2019), Dr Martin de Jode (UCL, 2019)).

Colouring Australia:

Dr Henry Peterson (University of New South Wales)
Alireza Shamakhy (University of New South Wales)

Colouring Bahrain:

Mohamed Ahmed, Back-end developer - part-time. Freelance.
Husain Aldurazi, Front-end developer - part-time. Freelance
Ali Naser, Development operation - part-time. Freelance.

Colouring Canada

Alireza Adli (Concordia University)
Christopher Gibbs (Concordia University)

Colouring Colombia: The District University of Bogotá:

Maria Camila Velez (Editor, Age)
Javier Alejandro Beltran (Editor, Location)
Jean Janer Villafañe Romero (Editor, Construction)
Bryan Felipe Urrego (Editor, type)
Ebliss Yissela Segovia Venales (Editor, Current use)
Jenny Gabriela Romero Sua (Editor, planification)
Angie Juliana Pirachicán (Editor, landscape)
Jeyson Santiago Gómez Gómez (Editor, team)
Laura Ximena Choconta Pardo (Editor, sustainability)
Wilson Daniel Buitrago Mendoza (Editor, Dynamics)
David Mateo Liévano González (Editor, Community)
Juan David Dallos Becerra (Editor, Size)

Colouring Germany:

Theodor Rieche(IOER, 2022-)

Colouring Greece:

Yiorgos Panagiotopoulos (National Technical University Athens, 2021-)

Colouring Indonesia: King's College London and the Institut Teknologi Bandung:

Dr Zara Shabrina, King's College London;
(Alumnae: Please add with date)

Colouring Sweden (Mälardalen University):

Jalil Karimi (Mälardalen University)

Colouring Lebanon (American University of Beirut):

Current: Mahdi Maatouk

Notes on appointing software engineering teams

Key findings re sustaining high quality teams

In addition to excellence in research software engineering, excellent working relationships between software engineers and non-technical researchers involved in content design are also critical. Platform development and management will involve painstaking incremental adjustment of interface details and subcategories, testing of diverse data capture methods, exploration of new data categories and types, tailoring of categories and interface to maximise accessibility for diverse audiences, as well as foreseeing and addressing data security and ethical concerns. Patience, helpfulness and a real interest in open data and in mutually supportive, cross disciplinary working are therefore key characteristics required by all members of CCRP teams.Key findings from Colouring London prototype development between 2016 and 2022 were as follows:

High quality research software engineering (RSE) input is integral to the success of all CCRP platforms;
RSE expertise and building stock expertise should be viewed as of equal importance in platform design;
RSE interest in open data, multidisciplinary collaboration, and communication, are vital. Not all engineers will interested in working with specialists from the humanities and the arts or in open knowledge systems. Interest in both as well as curiosity, patience, and flexibility are considered critical for platform success;
Reliable, high quality SE teams are hard to put together and very difficult to sustain if they are run outside academic departments. Good freelance engineers often want to experiment with a range of projects. Money is often not a major incentive for engineers interested in open data therefore additional funding is unlikely to solve the problem of SE skills/knowledge gaps/loss;
Ongoing change of SEs on CCRP teams is likely to occur causing interruption of work flows and time and knowledge loss. In-house academic software engineering teams are considered a more reliable source of high quality research software engineers but changing priorities of RSE teams over time should also be anticipated in planning;
High quality software engineering is expensive. The scale of funds required to be raised to pay for even small amounts of engineering time means that large amounts of highly experimental work on platforms are often quite hard to carry out. Co-working on core repoistory code is designed to reduce costs of engineering time for all partners;
Colouring platform hosts should be aware in advance of the importance of planning for continuity of engineering and in also developing research softare engineering teams networks at national level, across universities to support this process. CCRP PIs are reliant on excellence and helpfulness of SE teams. Without CCRP SE engineering team collaboration across and within countries, SE team management and continuity, appointment of high quality engineers in a highly competitive environment, is likely to be an extremely time consuming and difficult aspect of project direction for PIs.

Recommendations in relation to CCRP partner software engineering teams

Prioritisation of appointment and retention of research software engineers i.e. those working within academic research teams is strongly recommended over commercial appointments to maximise continuity and retain knowledge and advance research goals, and minimise costs of admin time appointing new engineers ad getting them up to speed. This approach also allows as much project funding as possible to be fed directly into research;
Early collaboration with academics and students involved in computer science within CCRP host institutions, or with their partners/specialist SE research institutions within countries, will allow for as much research software engineering (RSE) continuity as possible. This also allows as much project funding as possible to be fed directly into research;
Small-scale internal academic kick start funding for technical demonstration of platforms built by students/research engineers. 2 year f/t SE engineering support then ideally be sought to test and consult on platform and to level larger grants. However 0.5 fte would also likely be sufficient;
Take advantage of free one-to-one technical sessions and group sessions coordinated by the Turing and the University of Oxford designed to support demo set-up;
Encourage engineers to identify areas of particular interest to them and encourage/help make time for experimentation in these areas, within platforms, alongside other tasks.
Develop long-term relationships with individual engineers. Software engineers that have formed an integral part of set-up teams may, even if they move to other posts, continue to be interested in being part of the team as informal advisors to the projects, as and when suits them, and to potentially work on short or longer term collaborations in the future.
Developing an ongoing relationship with engineers working on open source platform and open databases for the public good such as OpenStreetMap is also important.

Software engineering jobspec recommendations

For a “Programmer”, “Software Developer” or “Research Software Engineer” who will be able to contribute to Colouring Cities software development, set up and customise a Colouring Cities site and engage with the research from a technical standpoint. Academic/research software engineers based in collaborating CCRP partner institutions, or collaborating with them are strongly recommended (see also below) to maximise skills and knowledge sharing, and continuity within software engineering teams.

Essential

Programming knowledge of JavaScript and/or TypeScript.
Web development experience, including knowledge of NodeJS, using Express and React or similar libraries for frontend and backend development.
Experience using Git and GitHub for version control.
Knowledge of setting up and maintaining cloud-based web applications (for example with AWS EC2 or Azure VMs and related services).
Commitment to working openly and collaboratively on the project.
Positive approach to and interest in a range of types of problem solving.

Desirable

Experience developing backend web applications using PostgreSQL, Nginx, NodeJS.
Interest in user experience (UX), frontend web and user interface (UI) design.
Interest or experience in working with geospatial data and interactive visualisation, particularly web-maps (tools like Mapnik, tileservers, Leaflet.js, MapboxGL.js).
Interest in open geospatial data generally (OpenStreetMap, government and public open data initiatives).
Ability to break down features into focussed tasks/stories and deliver incrementally.
Excellent interpersonal skills, easy to work with, patient.
Ability to work in an efficient and focussed way.
Interest in identifying ways to facilitate engagement with diverse audiences (including public and private sector organisations, schools, community groups and academics).

Current model for research software engineer engagement with GitHub Colouring London repository

Image Ed Chalstrey. Alan Turing Institute. 2022

Below are additional diagrams produce by Ed Chalstrey in 2022 showing the development flow for open code development and platform testing at the Phase 1 (2016-2022) to Phase 2 (2023 onwards):

Ed current development flow

ed long term development flow Image produced by Ed Chalstrey 2022

Overview of Colouring Cities platform set-up

Key steps

Commitment by CCRP participating academic institution to CCRP aims and protocols; CCRP partner legal team assessment to check wording and identify inclusion of additional legal disclaimers etc. as required; securing of set-up grant;
Inclusion of CCRP partner on the Alan Turing CCRP website page, and on open manual pages-including a dedicated country page;
Provision of access to free Colouring Cities domain name from the Alan Turing Institute;
Provision of acccess to co-work on Colouring Cities core colouring GitHub repository and documentation and set up environment;
One-one-sessions available from the Alan Turing Institute, as well as access to group CCRP technical meetings
Coordinate server access and credits;
Secure access to open building footprint data;
Work with PI Tailor to tailor subcategories to suit geographic location- translation and content PI to suit edit to edit text & CCRP partner institution
Maintain, manage and develop platforms
Work on core colouring code with CCRP partners and release country specific code
Co-work on papers and support research initiatives

Information for engineers transitioning from Colouring London to the new Colouring Cities colouring-core Configuration

Our aim is to maximise collaboration amongst the codebases represented in https://github.com/colouring-cities, as well as maximising sharing of standardised datasets. The CCRP strategy for 2023 involves a transition from CCRP partners forking and adapting Colouring London prototype to suit national/regional requirements, to a model allowing CCRP PI and engineering teams to co-work on Colouring Cities core code to increase efficiency, lower engineering costs for countries, and maximise skills and knowledge sharing

Updated below by Mike Simpson, CCRP Research Software Engineer for the Alan Turing Institute/Newcastle University (Feb 2023)

As we move from the Colouring London prototype to the expanding Colouring Cities Research Programme (CCRP), we have decided to make several changes to the colouring-cities organisation on GitHub to enable better collaboration between the various projects.

We do not expect these changes to significantly impact the existing “Colouring...” projects. However, we have detailed the changes in this document to minimise confusion and explain our vision for the CCRP moving forward. Colouring Core, The idea behind creating the ‘colouring core’ platform is to have a single base repository from which all other Colouring Cities projects are forked. That way, developers on all projects can share their features and innovations with the rest of the CCRP by folding those changes back into the Core platform.

Colouring London transition

The problem was that several projects were already forked from the Colouring London repository. For example, the two projects shown in the diagram below.

Introducing a new ‘core’ repository into this organisation structure would have led to additional work for the Colouring London team, and all the projects forked from Colouring London. Instead, we decided to go with a new structure, which involved renaming some of the Colouring Cities Repositories: • We renamed the ‘colouring-london’ repository to ‘colouring-core’. • A new ‘colouring-london’ repository has been created, forked from ‘colouring-core’. We appreciate that this may cause some confusion, but we decided it was the best way to move the project forward.

The new organisation structure looks like this:

This change puts the onus on us to make changes to the Core and London platforms, rather than on our partners to have to adapt to fit in with this new structure.

Moving Forward The new structure is intended to make easier to customize local platforms while minimising the divergence in codebases between project and allowing certain new features to be merged back into the core platform. Ideally, all CCRP projects should be forked from the new Core repository and should contribute any useful features back to the Core Platform.

We encourage all Colouring Cities projects to implement new city-specific things by modifying the colouring-core repository via pull requests and configuring it locally, rather than making changes locally and diverging from the core repository, where possible. The aim of the restructure is to minimise the amount of duplicate effort and city-speicifc code in each project, and to ensure that new developments and features can be shared with all CCRP members.

Please report and/or submit Pull Resuests about any London-specific things that you find in the code, so that they can be moved into the new configuration file system (see other document attached to this email for details).

Possible Problems We do not expect these changes to cause any technical problems, beyond some minor confusion. Projects previously forked from Colouring London automatically update on GitHub, and their relationship with the core repository will be preserved. There may be some issues, however. For example, you may have cloned the old ‘colouring-london’ repository using GitHub Desktop. In that case, the application may now point to the new ‘colouring-london’ repository instead of automatically updating to point to ‘colouring-core’.

Please ensure that any links and references in your code, documentation and applications are updated to reflect this new structure. Also, please double-check that you are using the correct repository before checking in any code.

Feedback If you experience any issues with the new organisation structure, feel free to give us feedback using the discussion features on GitHub or create an issue.

Colouring Cities core updates

As part of the transition from the Colouring London prototype to the expanding Colouring Cities Research Programme (CCRP), we have decided to implement some changes to make it easier for other projects to create and customise their own Colouring Cities platform. The first stage of this process has been the addition of a new config file, used to specify certain elements of the Colouring Cities interface. These changes have now been deployed on the new 'colouring-core' and 'colouring-london' repositories.

Using CC-Config

To customise the site, you will need to edit the file “app/src/cc-config.json” and modify the text strings (red in the example below) and the values (shown in green below) to customise the Colouring Cities interface.

For example, changing the "cityName" from "London" to "[Your City Name]" will replace the logo and other properties across the entire site to use your city name instead of London. You are also encouraged to update your githubURL to refer to your specific version of the project on GitHub. Also, please update the privacyStatement to inform users how their data will be stored (and what safeguards are in place).

More Features/Documentation Coming Soon

We will add new features and more information to the Colouring Cities Core Platform repository documentation as soon as possible. You can find this here: https://github.com/colouring-cities/colouring-core If you have any questions, comments or suggestions about this feature, feel free to use the discussion facilities available on GitHub. Using/Modifying CC-Config in your Colouring Cities Code For Developers, Software Engineers and advanced users, here is a more detailed overview of how the CC-Config features work. The CCConfig code is defined in app/src/cc-config.ts and contains all of the variables that can be used in your code.

The value of these variables are defined in JSON format in app/src/cc-config.json

And to use the cc-config parameters in your code, you will need to import the files using the following lines of code import { CCConfig } from '../../cc-config'; let config: CCConfig = require('../../cc-config.json')

Colouring Cities Configuration.docx

Colouring Cities Core Platform.docx

General information on platforms and resources available

Diagram showing operation and data schema

First iteration produced by Mateusz Konieczny Feb 2023

Diagram showing CCRP content structuring

Languages

The main language used is TypeScript, with Python scripts for some data preparation. The database uses PostgreSQL and repository also contains various SQL scripts used for setup.

Use of Colouring Cities domain name

The Alan Turing Institute provides CCRP collaborators with access to domain name at no cost, with city/country.colouringcities.org is available to all partners. (colouringcities.org was purchased for the project in 2019). We can also and point domain names using research institution info - i.e. colouringathens.ntua.gr - to the same site.

Server and storage requirements

Cloud services suggested for running a Colouring Cities application.

Virtual Machines

For Colouring London, we use Microsoft Azure and find that the “burstable” instances are cost-efficient because the site sees uneven usage and goes through relatively quiet periods.

B series: https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-b-series-burstable

For development, we use local desktops or laptops.

The current setup runs everything in a single virtual machine:

NodeJS application 

PostgreSQL database 

Nginx reverse proxy server

For internal demonstrations and testing, we have a “dev” virtual machine (B2s) which we only run for perhaps a few days in a month.

For deployment, we have a “staging” virtual machine (also B2s) which runs the latest version of the code, and can be used for testing before making an update live.

B2s specification:

2 cores 

4GB RAM 

30GB disk

For live, we have a “production” virtual machine (B4ms).

4 cores 

16GB RAM 

32GB disk 

additional 256GB disk attached for local storage of backups and data extracts, before moving these to long-term blob storage

Blob Storage

Blob storage (Azure blob storage, Amazon S3 Simple Storage Service) is used for long-term storage of database backups and data extracts. This is a pay-what-you-use service with no need to specify storage space or bandwidth up front, and no practical limits on storage capacity (up to 5PiB).

Pricing

The pricing calculator is useful for estimates: https://azure.microsoft.com/en-gb/pricing/calculator/

A similar setup to the one described above on Azure would budget at around GBP 150-200 per month. The virtual machines are the largest component of that cost – the smaller B2S instances should be sufficient to start with, and 40-60% discounts are available if you can commit to a contract for a year or three years.

Other cloud providers have very similar services for similar prices.

Translation tools ADD

Number of edits info

Information on edit numbers can most quickly be retrieved by CCRP teams from leaderboards (top 100, all time): https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcolouring.london%2Fleaderboard.html&data=05%7C01%7Cphudson%40TURING.ac.uk%7C40523e8883874885524308daef198821%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C638085190187922163%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1sq1ctZtHfI4m5YgSKQLcWltPLIvMmzmQV4LcmGwiLU%3D&reserved=0 and pop the numbers in a spreadsheet.

Once tagged edits should ideally be split on the Leaderboard into i) Open bulk uploads from official sources, 2) streamed form official sources, 3) manually entered/crowdsourced, 4) automated using inference.

Sign up, Personal data and data security for Colouring London

Separate sign ups are required for Colouring London editing, and for the Mailgun Disussion thread.
For Colouring London editing, users are actively discouraged from providing the CCRP with personal data. Users are not required to add personal information to use the site. The site is free to view. For those wishing to edit, only a username and password are required. The email address is optional. If provided this allows us to send the user an email to reset their password if they forget it, and in exceptional circumstances contact a user directly if it looks like they are misusing the site, for example to let them know if we plan to disable or remove their account.
The email address is stored in a database which can only be accessed from within the local network of the Colouring London application server, which is only accessible to select developers working on the project. Users of the Colouring London site can only access their own information (there is no "admin panel" or other kind of user with special access) and connect to the site using standard HTTPS encrypted communications. Documentation of crypt and gen_salt used can be found here.

Add info on: Can Turing provide email addresses to support communication with stakeholders e.g. [email protected] or [email protected] and/or [email protected]? Are there any alternatives to Mailgun that are recommended and free?

Backups

ADD

Downloads

ADD

Data accuracy and verification

Verification relates to current edits only. Where an entry is verified 5 times but then is replaced by a new edit/entry the old enrty and associated verifications will be stored under 'edit history' and verifications for the new edit only displayed (PH). However we should in fact have a system where if the current edit is overwritten with a new edit that matches an historical one, then the old verifications should also be restored with one extra verification also added for the original editor's entry. For further information on accuracy and verification see: ADD Manual link

Open Licences

The process of granting open licences is essential to give users of the platform, and GitHub, the right to share, use and build on open data and open code respectively. Recommendations for open data and open code licences were made in 2016 by Tom Russell.

For open data platforms, open licences are required for both data and for software. In terms of data sharing, owing to the fact that built environment stakeholders span the private, public and third sectors, open licences need to ensure third party use, to permit innovative collaborative working with commercial firms. All open data generated by Colouring London were therefore released under an OKF Open Database Licence (ODbL). This type of licence is also used by OSM (OpenStreetMap Wiki, 2021d). It is similar to the Creative Commons CC BY licence, but is designed for databases rather than individually collected records (Creative Commons 2021). It allows users to copy, distribute, transmit and adapt data, as long as Colouring London and its contributors are credited. To maximise openness, the licence requires that data can be altered or built on and that the result may only be distributed under the same licence type.

Information on Colouring London’s ODbL licence is provided on the Contributor Agreement page). The Contributor Agreement also states that data sources need to be included where possible to improve accuracy. This ODbL agreement, like the Code of Conduct, has to be accepted on the platform’s sign-up page, along with the Data Accuracy Agreement (see below), before users can edit data. The latter agreement was considered necessary by the author to protect host institutions against claims relating to data inaccuracy.

Copyright on Colouring London open code, released via GitHub, is shared among all contributors and licenced under a GNU General Public Licence (GPL), version 3 or later. This is a commonly used open software licence published by the Free Software Foundation. It was selected by Russell as it ensures that code can be reproduced at no cost and allows end users the freedom to run, share and modify the software, provided that the source is credited and that the code is released on the same or equivalent licence terms.

(Open data licences were first released in 2002 by Creative Commons, a non-profit organisation in the US (Creative Commons 2021). Creative Commons (CC) licences provide flexibility for authors (and can include options for non-commercial or commercial use). These also help protect users from copyright infringement. The most permissive is the CC BY licence. In 2014 CC licences were approved by the Open Knowledge Foundation (OKF) (Open Knowledge Foundation 2021b and 2021c), founded in the UK in 2004 to promote open knowledge. These conform to the OKF’s open requirement which states that data must be available at no more than reproduction cost, accessible in easily modifiable form, be able to be reused, redistributed and mixed with other data, and must not discriminate against fields of endeavour, people or groups (ibid.).