This tutorial explains how to configure Mailman to send received mails to ElasticSearch for indexing. ElasticSearch can then provide an alternative search to built-in Mailman search functionality.

• Install pyes Client
• Install Custom Mailman Archiver
• Configure Mailman External Archivers
• You're done!

Mailman is implemented in Python thus we will use Python ES client (in this tutorial we will use pyes ES client). If you are running Mailman you have already Python installed but you need to install Pyes too. To do this you can follow official pyes install notes. However, in this tutorial we will be installing trunk code directly from GitHub.

  # do as root

  # install pip http://pip.openplans.org/requirement-format.html
  easy_install pip

  # ensure you have python-devel or you get "fatal error: Python.h: No such file or directory"
  # if you do not have python-devel you can install(*) it via
  yum install python-devel
  # (*) - depends on your system

  # now download and install pyes
  pip install -e git://github.com/aparo/pyes.git#egg=pyes
  # You should see something like "Successfully installed pyes thrift urllib3"
  
  # go to the installation directory (/root/src/pyes/ in my case) and execute
  python setup.py install
  # that will install pyes into site-packages directory (/usr/lib/python2.7/site-packages/ in my case)

To verify that you installed pyes correctly you can run the following test from command line:

  python -c 'import pyes; print pyes.__file__'

It should output something like:

  /usr/lib/python2.7/site-packages/pyes-0.14.0-py2.7.egg/pyes/__init__.pyc

Now we need to download and install Mailman custom external archiver. Pyes contrib contains working template here: https://github.com/aparo/pyes/blob/master/contrib/mailman/archive_and_index.py

Download this file and install it into folder that can be red by process running Mailman, for example you can download it into Mailman bin folder (in my case it was /usr/lib/mailman/bin).

You need to tell Mailman to make use of this file by adding the following two lines to mm_cfg.py file:

  PUBLIC_EXTERNAL_ARCHIVER = '/usr/lib/mailman/bin/archive_and_index.py %(hostname)s %(listname)s'
  PRIVATE_EXTERNAL_ARCHIVER = '/usr/lib/mailman/bin/archive_and_index.py %(hostname)s %(listname)s'

Then restart Mailman.

(Note that you may need to recompile mm_cfg.py file on some systems first. Check mm_cfg.py or Mailman documentation for more details. In my case it was enough to execute /usr/bin/mailman-update-cfg)

Yes, you are done. Now all messages received by Mailman are indexed by ElasticSearch as well.

You should also modify the archive_and_index.py script and change default ES connection settings and/or other settings accordingly. Also you may want to elaborate the script a bit more as the code is not meant for production installation without prior customization.

This integration is built on top of the following resources:

• How do I invoke some process on messages as they are added to the pipermail archive
• Indexing mail right after delivery