Implemented regularizers

This page describes the features and cases of usage of the regularizers, that have already been implemented in the core of BigARTM library. Detailed description of parameters of the regularizers in the Python API can be seen here.

Examples of the usage of the regularizers can be found here

Smooth/sparse Phi

M-step formula:

p_wt \propto (n_wt + tau * f(p_wt) * dict[w])

Description:

This regularizer provides an opportunity to smooth or to sparse subsets of topics using any specified distribution.

To control the type of the distribution over tokens one can use the dictionary (dict), and the function f. dict is an object of Dictionary class, containing list with all unique tokens and corresponding list of values, which can be specified by the user. These values are dict[w]. f is a transform function, the derivative of the function under the KL-divergence in the source formula of the regularizer.

If the dictionary is not specified, all values will be 1. If is specified, and there's no value for the token in it, the token will be skipped. The f is const 1 by default.

Usage:

There're several strategies of usages of this regularizer:

• simply smooth or sparse all values in the Phi matrix with value n: create one regularizer and assign tau to n;
• divide all topics into two groups (subject and background), sparse first group to increase their quality and smooth second one to gather there all background tokens: create two regularizers, specify topic_names parameters with corresponding list in both ones and set tau in sparsing regularizer to some negative value, and in smoothing to some positive one;
• smooth or sparse only tokens of specified modalities: create one regularizer and specify parameter class_ids with list of names of modalities to be regularized;
• smooth or sparse only tokens from pre-defined list: edit the Dictionary object of the collection and change value field of the dictionary for tokens from this list should be set to some positive constant (remember, that it will be dict[w] in formula), and for other tokens value field should be set to 0;
• increase the influence on the small p_wt values and reduce on the big: define the f function by specifying the KlFunctionInfo object to kl_function_info parameter (don't forget, that f will be the derivative of this function).

All these strategies can be transformed, merged to each other etc.

Notes:

• smoothing in second strategy should be started from the start of model learning and be constant;
• sparsing should start after some iterations and it's influence should increase gradually;
• setting different values to value field for tokens from pre-defined list in pre-last strategy gives an opportunity to deal with each token as it is necessary.

Smooth/sparse Theta

M-step formula:

p_td \propto (n_td + tau * alpha_iter[iter] * f(p_td))

Description:

This regularizer provides an opportunity to smooth or to sparse subsets of topics in Theta matrix.

To control the influence of the regularizer on each document pass one can use alpha_iter parameter, which is a list with length equal to number of passes through the document, and the function f. f is a transform function, the derivative of the function under the KL-divergence in the source formula of the regularizer. If alpha_iter is not specified, it is 1 for all passes. The f is const 1 by default.

Usage:

There're several strategies of usages of this regularizer:

• simply smooth or sparse all values in the Theta matrix with value n: create one regularizer and assign tau to n;
• divide all topics into two groups (subject and background), sparse first group to increase their quality and smooth second one to gather there all background tokens: create two regularizers, specify topic_names parameters with corresponding list in both ones and set tau in sparsing regularizer to some negative value, and in smoothing to some positive one;
• increase the influence of the regularizer on later passes through the document: specify alpha_iter list with k values from smaller to greater, where k is a number of passes though the document;
• increase the influence on the small p_td values and reduce on the big: define the f function by specifying the KlFunctionInfo object to kl_function_info parameter (don't forget, that f will be the derivative of this function).

All these strategies can be transformed, merged to each other etc.

Notes:

• smoothing in second strategy should be started from the start of model learning and be constant;
• sparsing should start after some iterations and it's influence should increase gradually;
• fitting (with regularization or not) will be different in cases of ARTM.reuse_theta flag set to True or False.

Decorrelator Phi

M-step formula:

p_wt \propto (n_wt - tau * p_wt * sum_{s \in T\t}(p_ws))

Description:

This regularizer provides an opportunity to decorrelate columns in the Phi matrix (e.g. make topics more different), that allows to increase the interpretability of topics.

The influence of the regularizer can be controlled only with tau parameter.

Usage:

There're several strategies of usages of this regularizer:

• decorrelate all topics: one regularizer with tau, that should be tuned experimentally;
• in the strategy with two background and subject topics it is recommended to deal with each group separately: create two decorrelators and specify topic_names parameter in them, as it was done for Smooth/sparse regularizers;
• deal only with the tokens of given modality: set class_ids parameter the list with names of modalities to use in this regularizer.

All these strategies can be transformed, merged to each other etc.

Notes:

• this is a sparsing regularizer, it works well with general sparsing Phi regularizer;
• the recommendation is to run this regularizer from the beginning of the fitting process.

Label regularization Phi

ToDo(MelLain)

M-step formula:

Description:

Usage:

Specified sparse Phi

ToDo(MelLain)

M-step formula:

Description:

Usage:

Improve coherence Phi

ToDo(MelLain)

M-step formula:

Description:

Usage:

Smooth Ptdw regularizer

ToDo(AnyaP)

M-step formula:

Description:

Usage:

Topic selection Theta

ToDo(MelLain)

M-step formula:

Description:

Usage: