Name indexer - AtlasOfLivingAustralia/documentation Wiki
Name indexing happens during the "processing" phase of ingestion. If a customised/national checklist is required, a new name index can be created. Details of the required structure are here.
Using the playbook nameindexer-standalone.yml
Assuming you have an instance of ALA portal running, when you need to use your own name index, or update the existing name index, you want to run it separately so it doesn't potentially break the running service.
So as the tutorial of building a customised name index, we start from a vanilla vagrant ubuntu box.
$ cd ala-install/vagrant/ubuntu $ vagrant status
At this point, if you have a vagrant box running, consider destroy it and start a fresh one.
$ vagrant destroy $ vagrant up
Ensure the virtual box is up-and-running and you can login:
$ vagrant ssh
Now, run the playbook for creating the instance that builds nameindexes:
$ cd ../../ansible $ ansible-playbook -i inventories/vagrant nameindexer-standalone.yml --private-key ~/.vagrant.d/insecure_private_key -u root
Where the name sources are stored:
$ cd /data/lucene/sources
Observe the target folder before we do name indexing:
$ ls /data/lucene
It should have only 'sources' as a directory.
scp your zipped custom DwCA in to the directory
/data/lucene/sources, and expand the zip.
Now, to create a name index, for instance:
$ sudo nameindexer -dwca /data/lucene/sources/dwca-col-mammals
(As 23 Jul 14, the default owner of /data is root so you need sudo to run nameindexer.)
Notice that here the 'dwca-col-mammals' is a directory extracted from dwca-col-mammals.zip. If you are working on your own checklist in DwC-A format, make sure you extract it before fire the nameindexer. Do
$ nameindexer -help for more information.
Test search to see if the name index has been built successfully:
$ sudo nameindexer -testSearch "Macropus rufus"
And you should see:
Search for name ID: 6863103 GUID: urn:lsid:catalogueoflife.org:taxon:d9f7aefa-29c1-102b-9a4a-00304854f820:col20120124 Classification: "(Desmarest, 1822)",Animalia,Chordata,Mammalia,Diprotodontia,Macropodidae,Macropus Scientific name: Macropus rufus Authorship: (Desmarest, 1822) Rank: SPECIES Synonym: null Match type: exactMatch
That means you've got a working index.
Using your new nameindexer
After regenerating the name index, in your
/data/lucene directory, you'll see two new directories:
To use the new name index, you can zip these two directories and unzip and replace the same one on your production site in the VMs that use nameindex (that is,
Reindexing your data with the new name index
The next step would be to update the occurrence
solr index to use this new name index.
You can test the
biocache by processing a single record using
biocache process-single <Record UUID>.
Later you can re-process your data in the biocache with
biocache process -dr [druid] and re-index your data to see the changes
biocache index -dr [druid]. But probably you will need a full reindex of your data.
Supported checklist formats
@Todo : Text to add (maybe images as well)
The name indexing by default looks for
/data/lucene/sources/IRMNG_DWC_HOMONYMS to analyse homonyms. If you have alternative homonyms to detect against, run
-irmng flag and point to your own extracted homonym DwC-A.
To avoid obvious homonym indexing error, you can provide taxonomy hints in the Collectory-hub when you edit the metadata of data resources by providing a taxonomy hint. The URL would be
/collectory/dataResource/edit/[druid]?page=%2Fshared%2FeditTaxonomyHints. (Replace [druid] with your own.)
About using the GBIF Backbone
In you want to use the GBIF Backbone Taxonomy, you have to take into account that right now ALA nameindex uses a mandatory
scientificNameAuthorship and an optional
nameComplete, not like the GBIF backbone that uses a full scientificName following Darwin Core. Right now (Nov-2019) if you want to use the GBIF Backbone Taxonomy and avoid the duplication of authors in BIE (and other related issues), you can, for instance adapt the backbone using this gist to remove the
author from the
Or just download a GBIF backbone nameindex compatible with ALA
You can just download a GBIF nameindex compatible with ALA without following all these previous steps. For this:
- Put this in your inventories:
custom_namematching_url = https://datos.gbif.es/others/nameindex-gbif-backbone-sep-2021-lucene-6.tgz nameindex_to_use = custom
This it's now the default value in new generated or updated inventories, because if not the ALA nameindex it's used.
- Use this
tarin the first step of the bie-index admin tool (
DwCA Import): https://datos.gbif.es/others/gbif-backbone-sci-name-no-author-2021-09-14.zip it the one generated by the previous gist. You will save some hours even days of process time.
For more name indexing information read and follow the Guide to Getting Names into ALA.
This bie-index wiki page about the full reindex tasks is outdated but quite informative about the whole re index process.