Workers and parallelism - rebus-org/Rebus GitHub Wiki

Since everything in Rebus is async, one single worker can perform a ridiculous amount of work in parallel if that work can be awaited.

Therefore – to avoid doing too much work – the "max parallelism" concept is present, which puts a global max cap on how many messages to process in parallel.

You can configure those options by going

services.AddRebus(
    configure => configure
        .(...)
        .Options(o => {
            o.SetNumberOfWorkers(1);
            o.SetMaxParallelism(10);
        })
);

Which values to use for the settings depends on the type of work you want to perform. The following are some reasonable settings that maybe can give you some food for thought:

• Work is predominantly asynchronous – use a few worker threads and fairly high parallelism, e.g. o.SetNumberOfWorkers(2); o.SetMaxParallelism(20);
• Work is fast and synchronous – use a few worker threads and a matching parallelism, e.g. o.SetNumberOfWorkers(5); o.SetMaxParallelism(5); (although setting the parallelism higher will not affect anything in this case)
• Work is slow and synchronous – use more worker threads and a matching parallelism, e.g. o.SetNumberOfWorkers(15); o.SetMaxParallelism(15); (again: setting the parallelism higher will not affect anything in this case)
• Work must be performed in a synchronous manner, effectively serializing access to some particular resource (or to make an endpoint easier to debug during development) – use one single worker thread and a matching parallelism: o.SetNumberOfWorkers(1); o.SetMaxParallelism(1);

Since the parallelism setting puts an absolute upper cap on how many messages can be handled in parallel, it does not make sense to set the parallelism lower than the number of threads. It does not create any problems on the other hand though, it is just a waste of resources.

Defaults

Rebus will default to 1 worker thread and a max parallelism of 5.

Worker threads

Rebus will actually spawn the number of worker threads as dedicated threads. The threads will be used for polling the transport for messages.

Depending on whether the transport is asynchronous or not, the continuation after having received a message will either execute on the worker thread or on the thread pool.

Using dedicated worker threads is Rebus' default mode of operation.

TPL-based workers

Rebus also provides the options of using an alternative TPL-based worker factory, which will NOT spawn dedicated worker threads - instead, it will simply make a number of async calls to the transport.

For this to work properly, the transport must support an API for proper, asynchronous receive (which most modern transports do - but, incidentally, which the MSMQ transport does NOT!).

It can be enabled by going

services.AddRebus(
    configure => configure
        .(...)
        .Options(o => {
            o.UseTplToReceiveMessages();
        })
);

When using the TPL-based worker factory, the parallelism setting becomes redundant - the "number of workers" will define how many parallel receive operations will be commenced, and thus how many messages can potentially end up being handled concurrently.