installation operation - lmichel/vo-grimoire GitHub Wiki

Installation

This procedures are based on the serendib2023 setup parameters

Install Elastic Search

% sudo apt-get install apt-transport-https
% wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.0.3-amd64.deb 
% wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.0.3-amd64.deb.sha512 
% shasum -a 512 -c elasticsearch-9.0.3-amd64.deb.sha512 
% sudo dpkg -i elasticsearch-9.0.3-amd64.deb

The configuraton is located in /etc/elasticsearch

Make sure the secure pack is disable

sudo grep false /etc/elasticsearch/elasticsearch.yml 
#action.destructive_requires_name: false
xpack.security.enabled: false
sudo service elasticsearch restart

ES is launched at installation time. It must be tested first:

curl http://localhost:9200 
{
  "name" : "serendib2023.astro.unistra.fr",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xIYaRv68RYiDVjubnh-cLA",
  "version" : {
    "number" : "9.0.3",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "cc7302afc8499e83262ba2ceaa96451681f0609d",
    "build_date" : "2025-06-18T22:09:56.772581489Z",
    "build_snapshot" : false,
    "lucene_version" : "10.1.0",
    "minimum_wire_compatibility_version" : "8.18.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}

Install the software

Create the working directory

sudo  mkdir /data-local/dbvolumes/mail_indexation
sudo  chown laurent.michel /data-local/dbvolumes/mail_indexation

Install the software;

cd /data-local/git
git clone [email protected]:lmichel/vo-grimoire.git  MailIndexation
cd MailIndexation
mkdir logs

Adapt the configuration

more  serveur/python/launcher/config.json
{
  "download_uri" : "http://mail.ivoa.net/pipermail/",
  "elastic_search_url" : "http://localhost:9200",
  "mbox_dir": "/data-local/dbvolumes/mail_indexation",
  "reset_index": 0,
  "global_index": 1,
  "limit_request": 150,
  "mailing_lists": [
    {
   ...
}

grep serendib ./client/ivoa_web/www/index.html 
        elastic_search_url = "http://serendib2023.astro.unistra.fr/elasticsearch/"
            if( elastic_search_url.indexOf("serendib") > 0 ) {

Web Tuning

Edit /etc/apache2/sites-available/mail_indexation.conf:

   alias /esmail "/data-local/git/MailIndexation/client/ivoa_web/www"
    <Directory "/data-local/git/MailIndexation/client/ivoa_web/www">
        Options Indexes MultiViews
        AllowOverride None
        Require all granted
        Order allow,deny
        Allow from all
        Header set Access-Control-Allow-Origin "*"
    </Directory>

Enable this Apache configuration

% a2ensite mail_indexation
% systemctl reload apache2

Create virtual env

conda create -n elasticsearch python=3.12

Operation

First Run

Load and index manually the mails:

% conda activate elasticsearch
(elasticsearch)% cd serveur/python/launcher/
(elasticsearch)% python delete.py 
(elasticsearch)% python main.py

The delete operation must always be done first.

Test the Web application

connect http://serendib2023.astro.unistra.fr/esmail/

It should work...

Periodic update:

The mail indices must be updated daily by a cron job:

0 */4 * * * /bin/sh -c 'cd /data-local/git/MailIndexation/serveur/python/launcher && /usr/bin/python3 delete.py && /usr/bin/python3 main.py > /data-local/git/MailIndexation/logs/main.logs 2>&1'
⚠️ **GitHub.com Fallback** ⚠️