Flushing - RobertoPrevato/rolog GitHub Wiki

rolog has built-in support for log targets that flush messages in groups. For example, this is necessary to optimize the number of web requests when sending log records to a web api, or enabling bulk-inserts inside a database, instead of contacting external services for each log record.

FlushLogTarget is an abstract class, derived from LogTarget, implementing methods to handle groups of records and failures.

Below is an example of flush target class that sends log records to some web api, in groups of 500:

from typing import List
from rolog import FlushLogTarget, LogRecord


class SomeApiLogTarget(FlushLogTarget):

    def __init__(self, http_client):
        super().__init__()
        self.http_client = http_client

    async def log_records(self, records: List[LogRecord]):
        # NB: implement here your own logic to make web requests to send log records
        # to a web api, such as Azure Application Insights 
        # (see for example https://pypi.org/project/asynapplicationinsights/)
        pass

When logging a group of records fails, flush targets handle retries with configurable and progressive delays. By default, in case of failure a flush target tries to log records 3 times, using a progressive delay of 0.6 seconds * attempt number, finally falling back to a configurable target if logging always failed. Warning messages are issued, using built-in Warnings module to notify of these failures. For example, it is possible to configure a FlushLogTarget to store log records on the local file system, as a last resort when sending logs to an intended endpoint fails repeatedly.

These parameters are configurable using constructor parameters fallback_target, max_size, retry_delay, progressive_delay.

Parameter	Type	Description
max_size	int, default 500	The number of log records to store in memory, before flushing and calling log_records method.
max_retries	int, default 3	The number of max retries, when a call to log_records fails.
retry_delay	float, default 0.6	The number of seconds before a retry, when an attempt to send records to desired destination fails.
progressive_delay	bool, default True	Whether to increase the retry_delay, multiplying it by the attempt number.
fallback_target	LogTarget, default None	A fallback target to use when calls to log_records fails more than max_retries times.

class FlushLogTarget(LogTarget, ABC):
    """Base class for flushing log targets: targets that send the log records
    (created by loggers) to the appropriate destination in groups."""

    def __init__(self,
                 queue: Optional[Queue]=None,
                 max_length: int=500,
                 fallback_target: Optional[LogTarget]=None,
                 max_retries: int=3,
                 retry_delay: float=0.6,
                 progressive_delay: bool=True):

Flushing when application stops

Since flushing targets hold log records in memory before flushing them, it's necessary to flush when an application stops. Assuming that a single LoggerFactory is configured in the configuration root of an application, this can be done conveniently, by calling the dispose method of the logger factory.

# on application shutdown:
await logger_factory.dispose()