Search Service - OpenSlides/OpenSlides GitHub Wiki

The search service should provide at least the same functionality as in OS3: searching for models in a specific meeting.

The plan is to implement it independent of the autoupdate service and instead as a kind of presenter. This stems from the fact that the general use case is that a user wants to search for a model, spends a few seconds checking the result and then either clicks on one or closes the search, so it should not bother him if data changes while he has his search open. Optionally, a refresh button can be provided to update the search results manually. Another argument against using the autoupdate service is that the search is model-specific (different models have different fields that should be searched) and the autoupdate service is key-value based and does not have any model information.

The search service has the information which model fields should be searched (in OS3, this information was stored in the client). It accepts a request with a search string and an optional meeting id which should be search exclusively. Currently, OS3 offers filtering only based on the collection, which is given in the query like the following: "Motions: query". This can potentially be extended to include more filtering possibilities.

The current plan is to use an Open Source alternative to ElasticSearch as the backend since, if usable for our purposes, these will be much better optimized than our code. A simple, quick-and-dirty alternative would be to just fetch all data from the datastore and iterate over all models and check the fields. The alternatives are currently being evaluated and this page will be updated accordingly.

The search backend has to be filled with all data, this can be copied from the redis stream which publishes all updates. While updating the search data, HTML strings have to be parsed to be able to search in them and to also replace mutated vowels (äöü etc.). This could either be done by using a HTML parser in the backend, which could potentially be too slow, or by manually replacing/filtering characters and tags, which would be more error prone.

After the search backend returns the results, they must be send to the restrictor service to remove any models which the user is not allowed to see. This can only be done afterwards only on the condition that all fields that are searched are "basic" fields - meaning that if you have the permission to see the model, you automatically have permission to see these fields. If we at any point need "conditional fields" (e.g. a field which cannot be seen based on certain conditions, while the rest of the model is still visible to the user), we have to revisit this workflow! This could lead to errors where the search backend returns a model M based on field F while the user has permission to see the rest of M, but not F. Then M would be listed in the results without actually including the field which lead to the search hit (or, even worse, the user would see a field he does not have permission to see).

In the end, the search service must prepare the models for display in the client. Therefore, a kind of tree is needed to depict which model results in search results for another model (e.g. if a motion comment field containing the search query is found, the related motion should be displayed instead of the comment field). After resolving this, the search service should return a mapping from collections to a list of key-value-maps, which can then be directly displayed in the client. The general syntax could be like the following:

{
    "motion": [{
        "title": "Test motion",
        ...
    }],
    "user": [{
        "first_name": "test",
        ...
    }]
}

This is fully calculated in the search service. The fields differ for each model, but the format stays the same.