devnotes: first steps - DigitalCommons/open-data-and-maps GitHub Wiki

Boiled-down recipe for what I've managed to do so far.

• install prerequisites
• check out playground and open-data-and-maps
• build map-apps in the latter
• configure and deploy the map-apps
• build and deploy the data TODO

FIXME fill in docs/README.md

sudo apt-get install build-essential patch ruby-dev zlib1g-dev liblzma-dev git
sudo apt-get install ruby libzip
sudo gem install nokogiri
sudo gem install linkeddata

git clone ... open-data-and-maps -b development
git clone ... playground

Make sure our SSH config allows us to log in, and we have the currently hardwired aliases for logging in as joe and admin users.

Assuming your ssh public key has already been added to an account on sea-0 and sea-map, create a local ~/.ssh/config section as below (using your own user instead of $USER):

FIXME or use joe?

Host sea-0
Hostname sea-0.solidarityeconomy.coop
User $USER

Host sea-map
Hostname sea-map.solidarityeconomy.coop
User $USER

Host sea-0-joe
Hostname sea-0.solidarityeconomy.coop
User $USER

Host sea-map-joe
Hostname sea-map.solidarityeconomy.coop
User $USER

Host sea-0-admin
Hostname sea-0.solidarityeconomy.coop
User admin

Host sea-map-admin
Hostname sea-map.solidarityeconomy.coop
User admin

Make sure we can access the right accounts on sea-map and sea-0 (As per map-apps/data/docs/README.md "Authorising keys on the remote server") (Or roughly equivalently, as per README.md in the playground repo)

FIXME assumes use of joe? Logged on to sea-0 and sea-map, do:

sudo tee -a <~/.ssh/authorized_keys ~joe/.ssh/authorized_keys
sudo tee -a <~/.ssh/authorized_keys ~admin/.ssh/authorized_keys

There should be no build directory yet in open-data-and-maps/map-app. Neither should there be a configuration directory in open-data-and-maps/map-app/www/map-app

Choosing the variant test, because this is an experiment,

cd open-data-and-maps/map-app
make variant=test build # invokes configure target, then packs source into builds/test dir

This should create the build and configuration directories.

cd playground/mini-website

FIXME README.md is wrong, should use make deploy-dry-run

Edit MAP_APP_SRC_DIR := ../../open-data-and-maps/map-app/builds/newbridge/map-app in Makefile to reference test or whichever variant you used above.

make
make deploy-dry-run

Note: you can actually build newbridge map-app and deploy it to the playground directory as specified in the playground if you build map-app with variant=newbridge or any of the predefined ones

This is in the data subdirectory.

In your local map-apps repository:

cd data/co-ops-uk/2017-06

First you have to put in the details in the specified edition file that you are using. If you go inside the editions folder you will see all the edition .mk files that you can load as an edition. In this case:

nick@mee-pok:open-data-and-maps/data/co-ops-uk/2017-06$ ls editions/
final.mk  test2017.mk

These files contain the configuration and parameters for each configuration. Make sure to change the host names to the correct host names - In this case - sea-0-joe and sea-0-admin

(NOTE: at this point due to permissions for mkdir and overwriting files you should use sea-0-admin for both)

Dean has added an admin login for both Nick and myself (on sea-0) - he hopes this does not raise any issues. Be careful with admin as all permissions are granted without the need of sudo.

Then you have to standardize the files you want to upload in your database. You do this by collecting all tables and joining them into one, while doing some filtering, using the following command (after deleting generated-data if present):

make -f csv.mk edition=<name of edition you want to deploy>

(Note, when running the first time this will attempt to regenerate postcode_lat_lng.json, and in doing so will output some 404-not-found errors for unknown postcodes.)

After this step you need to generate a sameas.csv file which is used later to inject predicates into your RDF files telling the objects that they are duplicates to some other objects. To do this go to the dotcoops folder

cd ../../dotcoop/domains2018-04-24/

and first execute (after deleting generated-data if present)

make -f csv.mk edition=experimental

(Note, again you may see 404-not-found errors for the same reason)

After that open match-with-coops-uk.rb and at the bottom you will find a line:

COOPS_UK_CSV = "../../co-ops-uk/2017-06/generated-data/final/csv/uri-name-postcode.csv"

change the directory name final to the co-ops-uk edition you just generated, then run

ruby match-with-coops-uk.rb

After that you should have a generated sameas.csv file in the generated-data/experimental directory in dotcoops

Having completing this you can go back to the co-ops-uk/2017-06 folder and continue the process (i.e. here but see also similar instructions here)

make -f ../../tools/se_open_data/makefiles/generate.mk edition=test2017

!!But!! it crashes because of a 404 error:

open_url': <http://vocabs.solidarityeconomics.org/experimental/essglobal/standard-content/organisational-structure.skos>: Not Found(404) (IOError)

(Note: This file can be found in a browser via a redirect, but the script seems unable to.)

Currently what is happening, a script called standard.rb is called to load a locally hosted vocabulary at http://purl.org/solidarityeconomics/experimental/essglobal/standard/organisational-structure to generate data via ::RDF::Graph.load. The resource it is trying to get is not found (probably due to redirection or purl.org error?) If you go inside your editions/<edition>.mk file you can insert this parameter in your code

ESSGLOBAL_URI := https://w3id.solidarityeconomy.coop/essglobal/V2a/

to say that you are using a new version, which avoids the problem above.

You can try to hard code the requested url in place of the line that's causing the error but this leads to problems later in the triple store upload (not quite sure the error is coming from here but presumably for now).

If you successfully manage to build the data then you follow the instructions in the documentation to deploy the current build then build the triplestore queries and deploy the data to virtuoso (i.e. here)

grep RSYNC editions/2017.mk # check whether --delete option is present
make -f ../../tools/se_open_data/makefiles/deploy.mk edition=test2017 --dry-run # check what would be done
make -f ../../tools/se_open_data/makefiles/deploy.mk edition=test2017 # actually deploys!

If you manage to deploy and execute all the commands you should upload a new graph.

Current problem is the new graph leads to a 404 page...

Populating the triplestore (virtuoso database) is done like this:

make -f ../../tools/se_open_data/makefiles/triplestore.mk edition=test2017

This also uploads new data.

There was a step here concerning that 404 - Joe Billings helped us fix that, which (paraphrasing Dean) modifying sea-0-admin:/var/www/html/w3id.org/dotcooptest/.htaccess and create a new folder for the new triplestores. This adds a perma-url needed for the site to work.

• why two servers? any functional difference?

• are they used for production/dev? how can we ensure dev doesn't clobber production?

• should we log in as joe/admin, i.e. use the hardwired ssh aliases implied by the docs? Or as ourselves? -- answere; you can use your own user instead of joe, you both have the same rights. I have added both our public keys to admin so we should be able to ssh into it without a problem. And we are supposed to use it

• Do we need to update the hardwired MAP_APP_SRC_DIR := ../../open-data-and-maps/map-app/builds/newbridge/map-app to reference whatever variant we've configured, e.g. test?

• Are some resources and vocabularies missing? How has the update history been on that? Why is co-ops-2017-06 throwing a 404 when looking for the resource? purl.org vs w3 ? is coops-uk test2017 outdated, when deployed only an empty graph shows up (or is it even empty, it's just redirecting to a 404) checkout https://w3id.solidarityeconomy.coop/coops-uk/test2017/ virtuoso or https://w3id.solidarityeconomy.coop/dotcooptest/

• map-app variant config database naming convention?

• Backups

• why are map-apps kept/deployed on the database server? When deploying mapapp do you deploy the whole folder?

• What is OntoWiki?

• virtuoso db running on http instead of https?