Tutorial - salmito/leda GitHub Wiki

Leda Tutorial

Leda Main API Basics

This page presents an overview of the features of the Leda library, showing a few use cases.

The suggested way to use the Leda library is to add the following to the start of the Lua file that uses one of its functions:

require "leda"

Please note that this will define a leda global variable loaded with its standard API functions.

Stages are instantiated by the leda.stage API function which receives a Lua table with the stage’s definition and returns a stage reference to be used when defining the application graph.

The Lua table that defines a stage must contain a handler function that is used as the event handler of the stage. Each event coming for this stage will be passed to this function as its parameter. The following Lua code define a stage that receives a single event and prints it.

require "leda" --> load the 'leda' module

local s=leda.stage{ --> Create stage s
   handler=function(str) --> For each event call this function with 1 argument (str)
      print(str) --> Print str
      leda.quit() --> Quit after printing first event
   end
}
s:push("Hello world") --> Put the string on the event queue of 's'
leda.graph{start=s}:run() --> Create and run a graph with the stage 's'

Graphs are created by the leda.graph(PARAM) function call, where PARAM is a table defining the connectors of the created graph. This allows to describe a graph in form of a Lua table. The name of the graph is either provided as PARAM[1] or PARAM.name. All other elements with numerical index must contain constructors for the creation of connectors. Leda stages objects have the connect method to deliver appropriate constructors for connecting stages inside a graph definition. A optional 'start' attribute define the first stage of the pipeline graph.

Example:

... Create stages
local graph=leda.graph{"Graph Name",
	start=stage1, --optional
	stage1:connect('output1',stage2),
	stage1:connect('output2',stage3),
	stage2:connect('output1',stage4),
	stage4:connect('output1',stage5),
	stage4:connect('output2',stage6),
	stage5:connect('output1',stage1)
}
graph:plot("file.png") --> Requires luagraph to be installed

Produces the following stage graph:

Defining stages

Stages are instantiated by the runtime to process events placed in its correspondent queue. Each instance has a volatile state and a user-level thread abstraction (a coroutine). Instances of the same stage do not share state and are created and destroyed as needed by Leda to process incoming events. The maximum number of active instances is controlled individually for each stage at runtime, albeit, the programmer may define a serial flag to create single-instance (with persistent state) stages.

Instances of a stage have independent state, which are initially loaded only with its event handler (i.e. with no Lua standard libraries loaded). In order to initialize its state, the optional init function of a stage is called upon the creation of a new instance.

require "leda"
local stage=leda.stage{ --> Define a new stage
   name="Example stage", --> optional name (used in tostring)
   init=function()
      ... --> Code to initialize its instances
   end,
   handler=function(PARAMS)
      ... --> Event handler definition loaded in each of its instance
   end
}

Stage Communication

Leda offers a single leda.send primitive for handlers to emit events to other stages. Events are chunks of data that are delivered to the connector bound to a provided output port.

Stages can freely exchange values of the following Lua types in all types of connectors: nil, number, string, function (with its closure), and table. If necessary, a connector can be marked as local and may be used to deliver values of userdata type. Coroutines cannot be passed to other stages.

The next example contains a stage that generates random numbers, sends them to its value output port and sleeps for a period of time before generating the next number:

--file rand.lua
require "leda"

return leda.stage{
   handler=function(period) 
      while true do --> Loop
         local data=math.random() --> Generate a random number
         leda.send('value',data) --> Send data to 'value' port
         leda.sleep(period) --> Sleep for a period of time
      end
   end,
   init=function()
      require "math" --> Load the math library inside the current instance of this stage
      math.randomseed(leda.gettime()) --> Initialize the random seed
   end
}

Below we have a simple application that prints the values emitted by the previously defined rand stage.

require 'leda'
local rand=require 'rand' --> load the previously defined 'rand.lua' file

local printer=leda.stage{
   handler=function(...) 
      print(...)
   end
}

graph=leda.graph{
   rand:connect('value',printer) --> Connect the 'value' port of rand stage to the printer input queue
}

rand:push(1) -- stimulate the rand stage with a period of 1s

graph:run()

Stages can check if a required port is properly connected by defining a bind function. The following example shows a slightly modified version of the rand stage which assures that the value port is connected prior to the execution of the stage graph:

local rand=require 'rand' --> load the rand.lua file

function rand:bind(output) --> define a bind function for it
   assert(output.value,"Value port must be connected") --> Assures that 'value' port is tied to a connector
end

Handlers may receive values from its closure just as any Lua function. This is particularly useful for capturing command line arguments inside stages, for example:

local a="This string will be transported to each instance of the stage that uses it"
local command_line_param=arg[1]
stage=leda.stage{
   handler=function()
      print(a) --> will print the string 'a'
      if command_line_param then
         ... --> Command line argument exists
      end
   end
}

Clustering stages and running a graph

In Leda, different portions of the stage graph may be mapped on different OS processes that are running on different hosts. In such cases, events exchanged by different processes are serialized and delivered through the network according to the current graph configuration.

There are two basic types of connectors that can be used for binding stages of a graph: the local connector type requires that both communicating stages must always reside on the same cluster, thus being mapped to the same OS process. This connector type allows stages to exchange process related events, such as file descriptors, sockets or memory pointers (a userdata in Lua). The stages that are bounded through a decoupled connector (default case) may be on different clusters, therefore only serializable data can be passed through them.

A graph may be partitioned into several clusters, and each one must be mapped to exclusive OS process before the execution. A graph partition is created by calling the part method of a graph object.

The following example reads and prints the '/etc/passwd' file of two hosts (host1 and host2) on the terminal of a third host (host3):

require "leda"

local readfile1=leda.stage{
   handler=function(file)
      local io=require 'io'
      local f=io.open(file,"r")
      local str=f:read("*all")
      leda.send('text',str)
   end
}

local readfile2=leda.stage(readfile1) --clone the stage1 definition

local printer=leda.stage{
   handler=function(...) 
      print(...)
   end
}

graph=leda.graph{
   readfile1:connect('text', printer),
   readfile2:connect('text', printer)
}

readfile1:send('/etc/passwd')
readfile2:send('/etc/passwd')

graph:part(readfile1,readfile2,printer)
graph:map('host1','host2','host3')

graph:run() --> The run method of a Leda graph object starts its execution.