Duplicating Docker Influxdb Portals - earthcubeprojects-chords/chords GitHub Wiki

This is the procedure for duplicating a CHORDS portal that is running Influxdb. This assumes that a new portal has been provisioned and is running.

There are three main steps:

  1. Portal configuration: Duplicate the portal configuration, using the CHORDS configuration export/import function.
  2. Influxdb: Duplicate the Influxdb database, using the so-called 'line' protocol. The line protocol is used because we want to copy only the data in the database, rather than the complete database configuration with all of its metadata, shards, etc.
  3. Grafana: Duplicate the Grafana dashboards.

One of the cool things about this procedure is that you can can keep repeating the influxdb transfer from the old portal to the new one. The existing data will just be replaced in the new portal, and any newer data added. It's very useful if you want to test a new portal with existing data, and when you are satisfied that it is working, just do a final transfer even after switching data feeds to the new portal.

1. Portal Configuration Migration

  • Export the configuration from the old portal (via the configuration page).
  • Import the configuration into the new portal.

2. Influxdb Database Migration

  • Run the influx_inspect command on the old influxdb container, to create the database dump in line format, and copy it from the container to the host:
> docker exec -it chords_influxdb influx_inspect export -database chords_ts_production  \
  -datadir /var/lib/influxdb/data -waldir /var/lib/influxdb/wal -compress -out chords-influxdb.line
writing out tsm file data for chords_ts_production/autogen...complete.
writing out wal file data for chords_ts_production/autogen...complete.
> docker cp chords_influxdb:chords-influxdb.line .
> ls -lh chords-influxdb.line 
-rw-r--r-- 1 root root 2.9M Apr 19 15:26 chords-influxdb.line
> docker exec -it chords_influxdb rm chords-influxdb.line  # remove the dump file from the container

  • Copy chords-influxdb.line to the new portal, using whatever method is appropriate.

  • Copy the line data file to the influxdb container on the new portal, and run the influxdb command to populate influxdb:

> docker cp chords-influxdb.line chords_influxdb:/
> docker exec -it chords_influxdb influx -username admin -password <very_secret> -database chords_ts_production \
  -import -path=/chords-influxdb.line -compressed -precision=ns
2017/04/19 15:58:16 Processed 100000 lines.  Time elapsed: 1.062434386s.  Points per second (PPS): 94123
2017/04/19 15:58:17 Processed 200000 lines.  Time elapsed: 2.145215863s.  Points per second (PPS): 93230
2017/04/19 15:58:18 Processed 300000 lines.  Time elapsed: 3.18700255s.  Points per second (PPS): 94132
2017/04/19 15:58:19 Processed 400000 lines.  Time elapsed: 4.250657365s.  Points per second (PPS): 94103
2017/04/19 15:58:20 Processed 500000 lines.  Time elapsed: 5.28452076s.  Points per second (PPS): 94615
2017/04/19 15:58:21 Processed 600000 lines.  Time elapsed: 6.353640307s.  Points per second (PPS): 94434
2017/04/19 15:58:22 Processed 1 commands
2017/04/19 15:58:22 Processed 691524 inserts
2017/04/19 15:58:22 Failed 0 inserts
> docker exec -it chords_influxdb rm chords-influxdb.line  # remove the dump file from the container

3. Grafana Dashboard Migrations

  • Save the configuration for each dashboard in Grafana on the old portal, via Share Dashboard->Export->Save to File
  • Log in as admin on the new portal's Grafana.
  • Import each saved dashboard json file via Dashboards->Import->Upload .json file

Other

  • The influxdb dump is fast and efficient. On tzvolcano with 18M samples, it took about 20s to create the compressed line file, which was only 58M.