Data storage - akvo/akvo-core-services GitHub Wiki

####Requirements

  • proven technology
  • Minimal maintenance
  • Scales well
  • Geo-aware functionaly

####Technology chosen

  • Postgres
  • PostGIS for geo-aware functionality

####Considerations

  • Because of the complexity of the relational-type data the Akvo systems hold, it might be interesting to consider triple-store principles. An interesting approach to facilitate multi-tenancy is taken by Force.com, explained in this documents: document 1 and document 2, and explained here.

####Different types of data There are two different types of data that a data system needs to deal with:

  • Relational type data - for example, project data. Characterised by having a lot of different kinds and fields, for example a survey or a project.
  • Raw data - streams of facts, for example answers to questions, or project updates.

For the second type of data, an immutable data approach makes sense for the following reasons:

  • simplicity: the dataset is a an ever-growing list of facts
  • the dataset is queryable at any time in its history
  • the data is human fault tolerant
  • the data storage and query processing layers are separate
  • simplicity: no need for index / update, only append.