Monitoring example

Prerequisites

You need following tools installed:

• docker
• docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

• Create virtual environment and activate it (eg. python -m venv venv && ./venv/bin/activate)
• Install required packages pip install -r requirements.txt
• Run python prepare.py for downloading datasets

Monitoring Example

Starting services

To start all required services, execute:

docker compose up

It will start following services:

• prometheus - TSDB for metrics
• grafana - Visual tool for metrics
• mongo - MongoDB, for storing raw data, predictions, targets and profile reports
• evidently_service - Evindently RT-monitoring service (draft example)
• prediction_service - main service, which makes predictions

Sending data

To start sending data to service, execute:

python send_data.py

This script will send every second single row from dataset to prediction service along with creating file target.csv with actual results (so it can be loaded after)

Batch Monitoring Example

After you stop sending data to service, you can run batch monitoring pipeline (using Prefect) by running script:

python prefect_example.py

This script will:

• load target.csv to MongoDB
• download dataset from MongoDB
• Run Evidently Model Profile and Evidently Report on this data
• Save Profile data back to MongoDB
• Save Report to evidently_report_example.html

You can look at Prefect steps in Prefect Orion UI (to start it execute prefect orion start)

Bonus: MongoDB viewing with Mongo Express

Use the compose file below to tweak the original

version: '3.1'

services:

  mongo:
    container_name: mongo-dev
    image: mongo
    volumes:
      - ~/dockerdata/mongodb:/data/db    
    restart: on-failure
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
      MONGO_INITDB_DATABASE: auth
    networks:
      - my-mongo

  mongo-express:
    image: mongo-express
    restart: on-failure
    ports:
      - "8081:8081"
    environment:
      ME_CONFIG_MONGODB_SERVER: mongo-dev
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: example
      ME_CONFIG_BASICAUTH_USERNAME: admin
      ME_CONFIG_BASICAUTH_PASSWORD: tribes
    networks:
      - my-mongo
    depends_on:
      - mongo
networks:
  my-mongo:
    driver: bridge

Viewing Evidently Report HTML Output

Viewing the Grafana Dashboard