How luigid works - StevenRing/luigi GitHub Wiki

luigid is the concert master or conductor responsible for the orchestration of each luigi run. luigid executes as an independent daemon process and offers services to luigi task-executing workers which contact luigid via sockets.

Reviewing the luigid source code note that it accepts a number of parameters and then executes the run() method of the luigi server module (which will be daemonised depending on the setting of the background parameter).

The run() method first instantiates and loads a CentralPlannerScheduler and then constructs a Tornado HTTPServer with associated RPCHandlers designed to service the requests made by various clients (including Workers and human users via a broswer).

Key to the operation of the various RPCHandlers is the RemoteScheduleResponder which delegates request to the CentralPlannerScheduler. The protocol used by the workers and the CentralPlannerScheduler is exposed by the client proxy and server objects described in rpc.py

State maintained by CentralPlannerScheduler

A brief look at CentralPlannerScheduler shows the planner maintains a list of all active Workers and the Tasks that they have registered with the central planner. Workers register the tasks as they are discovered via a rpc call to add_task().

The CentralPlannerScheduler is responsible for allocating all Tasks to Workers. When a Worker is idle it requests allocation of a Task via a rpc call to get_work(). The scheduler will find the next PENDING task (know to the worker) selected on the basis of priority and time since initial registration. The Task is marked as RUNNING.