Gbif Spain Production environment - AtlasOfLivingAustralia/documentation Wiki

GBIF.ES Production infrastructure

This document tries to document the IT infrastructure of the Spanish Atlas accessible at datos.gbif.es.

Our main infrastructure is hosted in the CSIC-IFCA datacenter.

You can get some similar inventories from https://generator.l-a.site/5332d9ba-97d5-4940-b262-c29c1703968d (press Hamburger top menu > Make a copy > Edit the copy to your needs).

Components

URL Component
datos.gbif.es main page
colecciones.gbif.es collectory
registros.gbif.es biocache-hub
registros-ws.gbif.es biocache-service
biocaches-store-0.gbif.es biocache-cli, cassandra, jenkins
biocaches-store-1.gbif.es cassandra
index.gbif.es solr
espacial.gbif.es spatial, geoserver, layer-service
regiones.gbif.es regions
listas.gbif.es species lists
especies.gbif.es species
especies-ws.gbif.es species service
logger.gbif.es logger
imagenes.gbif.es images
auth.gbif.es Auth CAS service
demo.gbif.es demo site

Instances and components diagram

GBIF LA infra

Branding

Our custom branding is in this github repo.

Backup system

We do the backups using backuppc software. It's not secure to backup database directories where the databases are running. To avoid this, be use ninjabackup to backup our mysql and postgres databases periodically. Later backupppc backup the dumps stored by ninjabackup in /var/backups.

Some storage stats:

Monitoring

All our services are monitorized using icinga externally.

We also use mmonit internally to monitor that the basic services (ssh, apache2, tomcat, mysql, etc) are running correctly and restart then if not.

We use ELK (Elasticsearch, Logstash/Filebeat & Kibana) for logs agregation, indexing, and analytics. We don't store the logs for more than a week. This is very useful for us to do bugs and error hunting.

We also use netdata to analyze metrics of resources usage and other data (CPU, RAM, disks, network, monit status, etc). Useful for tunning resources, diagnostic, etc.

We collect js errors in browser clients using sentry. This can be useful also for collect java exceptions & errors but ALA software should be patched for that.

Our servers are updated automatically via the debian package unattended-upgrades.

All this software is managed via ansible using an internal ansible extras repo (TODO: make this repo more generic and share it with other portals).

Demo & test site

We also maintain a demo test site in https://demo.gbif.es.