Sample And Index - AtlasOfLivingAustralia/documentation Wiki

Sample and Index against layers

Introduction

In this page we document how to:

Steps

We'll sample and raster data against some added layers.

  1. Check the configuration for biocache-store is pointing to sampling URL:
    1. $ /data/biocache/config
    2. $ more biocache-config.properties
  2. Look for spatial.layers.url=http://spatial.l-a.site/ws/fields
    1. You can do this by connecting with ssh to the livingatlas-demo server you are using and issue command biocache config | grep -e ".*layers.*url"
  3. Load a DwCA into the collectory. If you are just testing, please choose a small dataset (<50k records) just for speed, preferably Mammals (as this affects a later step of the documentation with taxonomy).
    1. For IPT users:
      1. Start here: http://collections.l-a.site/admin/
      2. Create a data provider and point at IPT instance by setting website URL to IPT URL e.g. https://ipt.gbif.es
      3. Click “Update data resources” button
      4. Note: check the unique fields. Typical values are catalogNumber or occurrenceID. The default is catalogNumber
      5. Find a UID e.g. ´dr123´ to load
    2. For Non-IPT users:
      1. Start here: http://collections.l-a.site/admin/
      2. Create a data resource
      3. Upload your DwCA
      4. Note: check the unique fields. Typical values are catalogNumber or occurrenceID. The default is catalogNumber
  4. Load a DwCA into the biocache using command line tool
    1. Use the command biocache load dr123
    2. Validate the data has been loaded using Cassandra command line tool. Use the tool cqlsh on the command line.
      1. Connect to occ keyspace using use occ;,
      2. Run select * from occ;.
  5. Process the data resource - Use the command biocache process -dr dr123
  6. Sampling - Use the command biocache sample -dr dr123
  7. Indexing - Use the command biocache index -dr dr123
  8. Test the indexing was successful by:
    1. Viewing the SOLR admin console: http://index.l-a.site:8983 See solr admin interface to tips to access this.
    2. View the results in biocache services
      1. http://biocache.l-a.site/occurrences/search?q=*:*
    3. Test with an Area Report in the Spatial Portal
      1. Search for Gazetteer Polygon e.g. “Queensland”
      2. Tools > Area Report - and follow wizard Successful sampling/indexing depends upon specific details being properly applied in sequence. It is easy to miss or bungle a step, and it can be hard to tell which one caused a problem. To avoid those headaches, testing the outcome of each step can help to troubleshoot. Also, some background information about system configuration provides an overview of the process, and will hopefully help to debug issues as they arise. To that end, a synopsis:

Troubleshooting