The ARIEL Codebase for Developers - The-Franklin-Institute/ARIEL_Builder GitHub Wiki
ARIEL Codebase
This document outlines the basic structure of the ARIEL Builder software.
Overview
File Structure
How It All Works
Advanced Topics
Known Issues with Graphics
Recommendations for A New Version
Overview
ARIEL Builder is written for Mac OS X in Python 2.6. It makes use of openFrameworks version 0062. Using a tool called SWIG, openFrameworks' C++ tools were wrapped into Python. In addition, a C++ build of OpenGL was wrapped up with it (as distinguished from something like PyOpenGL, which is a significantly slower Python implementation of OpenGL).
ARIEL Builder's graphics and event loop run within openFrameworks. OpenFrameworks itself runs on OpenGL; this means that all oF tools (with some exceptions) are accessible and OpenGL calls can be mixed in.
As you will soon discover, running a Python application entirely on a SWIG-wrapped C++ library poses some problems. The reason for this decision was to leverage both the speed of openFrameworks– for which there is no Python equivalent– and the dynamic nature of Python.
File Structure
The latest build of the software is contained within ARIEL 2.0/App. The ARIEL 2.0/Examples folder contains an incomplete selection of sample sketches as well as a few advanced prototypes from the ARIEL research project. All of the python scripts live in ARIEL 2.0/App.
There are four files that comprise the openFrameworks/OpenGL library:
_openframeworks.so - the compiled binary file containing the actual library
openframeworks.py - the Python wrapper file generated by SWIG
libfmodex.dylib - compiled library for FMod, a set of audio tools used by openFrameworks
libGLee.so - an OpenGL extension-loading library required by openFrameworks
When using the ARToolkit tools (packaged with this build of openFrameworks), LogitechPro4000.dat is required.
The main application files are builder.py, node.py, player.py, and App/python/lib/python2.6/site-packages/psl/gui.py. More on their functions in the how it all works section.
Color palettes, short descriptions for nodes, and several utilities are broken out into different files within the App folder.
The python folder contains a 32-bit (arch i386) build of Python 2.6 (required to use this build of openFrameworks) along with several libraries (NumPy v1.2.1, Serial v2.5-rc2, and a custom library called PSL). This accounts for the ~275mb size of the software package.
The User Nodes folder contains user-created ARIEL nodes. Upon launching ARIEL Builder, a script looks for valid nodes in this folder and imports them into the Builder environment.
How It All Works
Environment
Event Handling & Object Management
Nodes
Save & Undo
Loading Sketch Data
Player
Compiler
Environment
The main event loop can be found in builder.py. This script must be launched using the Python build that comes with the software package; arielbuilder.command is a shell script created for this purpose. Note that ARIEL 2.0/ARIEL Builder.app is simply an AppleScript that launches arielbuilder.command.
When builder.py is launched, an instance of the ArielBuilder class is created and passed into ofRunApp(). All of the environment GUI initialization happens within the function ArielBuilder.setup.
There's a very important file hidden deep within the structure of the Python build: App/python/lib/python2.6/site-packages/psl/gui.py. This file contains the boilerplate GUI code necessary to create (almost) all of the interactive objects within the Builder environment. This file is imported into builder.py; it contains a global variable called manager that represents a PSGObjectManager, the controller that maintains references to all of the GUI objects in the environment.
The environment's GUI is broken up into three major parts: the NodePallet, the gui.MenuBar, and the gui.PropertyDrawer. The NodePallet contains the menu by which nodes are created; within it, multiple NodeCategory objects store NodePalletIcon objects that represent each possible Node. When clicked, a NodePalletIcon will generate a Node (more on those later).
Event Handling & Object Management
All GUI-related classes inherit the gui.PSGObject base class. There's a neat little operation within PSGObject.__init__(parent=None) that looks like this:
if not self.parent:
global manager
manager.addObject(self)
else:
self.parent.children.append(self)
If a PSGObject is instantiated without being given a parent, it will automatically add itself to the PSGObjectManager. The only objects with parents are those that belong in a hierarchy: menu bar -> menu item -> menu entries, etc. This ensures that all GUI elements are kept in the same place. If/when creating new GUI elements, make sure they inherit the PSGObject class. There are several higher-level classes to choose from, depending on the desired functionality: Button, Label, and Window.
The openFrameworks base app contains several event functions that we can expose: keyPressed, mouseMoved, mouseDragged, mousePressed, mouseReleased, windowResized, and the main update and draw loops. Most of the mouse events are forwarded to manager, the PSGObjectManager that contains all of the GUI elements. Key events are forwarded to gui.focus, an instance of gui.FocusManager that determines where to forward key events depending on what is being interacted with.
Nodes
This document describes the Node class as it relates to the ARIEL Builder environment. For an in-depth look at how nodes work and how to create new ones, see the Creating New Nodes section of the wiki.
Given the complexity of the interaction with Node objects, there is a large amount of boilerplate GUI in place to expose meaningful interaction events. gui.Button in App/python/lib/python2.6/site-packages/psl/gui.py lays the groundwork. The bulk of it, however, is done within App/node.py.
Before getting into the specifics, let's look at the broad strokes. A Node object is comprised of a body, some inputs, and some outputs. Dragging an output (represented as a little downward-facing triangle) exposes a cord. Plugging an output into an input creates a connection (assuming the data types are compatible- more on that later). Once a connection is in place, it can be undone by grabbing the output and pulling it away from the input.
The Node class inherits the gui.Button class, giving it a body that is draggable. Node contains inputs and outputs arrays which store NodeInput and NodeOutput objects, respectively. A global ConnectionManager object called connector stores the connection information, however the NodeOutput class is responsible for drawing the cord that connects an output to an input.
The NodeInput and NodeOutput classes are abstract and are not used at the concrete node level; instead there are different input and output types such as Number, Matrix, Image, Polygon, GlyphData, VectorList, etc. that inherit NodeInput and NodeOutput.
A Node can have editable parameters e.g. variables, filepaths, drop-down menu selections, etc. that can be edited in the ARIEL Builder environment. The functions Node.setParameterDict() and Node.getParameterDict() are used to turn any parameters into a dictionary of key/value pairs upon request. When saving a sketch to a file, for example, Node.getParameterDict() returns a dict that stores all of the current parameter values. This is covered in detail in the Creating New Nodes section.
Interaction
A Node object itself only responds to being dragged or double-clicked within its square body. Node.handleMouse checks for mouse events on the inputs and outputs first. If a NodeOutput catches a mouse click, it responds by telling its Arrow object to draw a cord to the mouse location. If an output is being dragged and the mouse is released over an input, the output calls connector.add(self, input). Connector stores pairs that describe connections between nodes as ConnectionManager.connections[index] = (NodeOutput, NodeInput).
Double-clicking a node reveals the Property Drawer, where that particular node's properties– if it has any– can be edited. The Node.propertyWindow doesn't have anything in it by default. It's up to the concrete classes that extend Node to populate their propertyWindow with relevant menu items.
Concrete Node Classes
Concrete classes extend from the Node class, e.g. class Camera(Node). There are a host of functions and objects to use depending on the features of the node.
For a detailed look at how to create a new node, see the Creating New Nodes page.
Save & Undo
File saving uses the Python cPickle module. The ArielBuilder.exportGraph function in builder.py handles the process. It begins by taking the gui.manager.objects list, filtering it to contain only Node objects, and extracting the relevant data about each node by calling its Node.getExportData function.
def getExportData(self):
inputs = map(lambda o: (o.label,id(o)), self.inputs)
outputs = map(lambda o: (o.label,id(o)), self.outputs)
prams = self.getParameterDict()
layout = (self.x, self.y, self.w, self.h)
return `self.__class__`,inputs, outputs, prams, layout
Therefore the data being saved is a list containing the classname, inputs (and their unique class id), outputs (and their class id), parameters returned by Node.getParameterDict, and the layout information of each Node in the scene.
After the nodes have been saved, node.connector.getExportData() is called, which returns a list of all of the connections in the form (input_id, output_id).
Calling cPickle.dumps(data) on the final array returns a heavily-compressed and unintelligible string that can be unraveled later to recreate a scene.
To implement Undo functionality, gui.undo represents a gui.UndoManager which is given a cPickle string every time an undo-able action is performed. When undo is called the last recorded state is 'unwrapped' using cPickle.load(data).
This is not an ideal solution, as it does not currently take into account Property Window editing.
Loading Sketch data
Just as the state of the environment can be distilled into an array, the 'unwrapping' process can turn that array back into a set of nodes and connections. This process is contained within builder.py/ArielBuilder.loadGraph(). Using the classname stored for each node in the cPickled array, the eval() function is used to instantiate the node again. eval can evaluate a piece of python code, so the classname is fed in as a string along with "()", where it is then populated with all of the other data:
try:
newNode = eval(classname + "()")
# ...
newNode.x, newNode.y, newNode.w, newNode.h = layout
newNode.setParameterDict(params)
During this process, a small operation saves each input and output to a dict called index, along with a reference to the node that the input or output belongs to:
for i in inputs+outputs:
index[i[1]] = (newNode,i[0])
This rather cryptic line of code says:
for every input and output in this particular node:
the location "input/output id" in the dict "index" = (reference to its node, "label of the input/output")
Once all of the nodes have been created the connector data is analyzed. This is somewhat complicated because even though we stored the unique input and output id()s, those ids will be different now because we've actually created new instances of all of the nodes, inputs, and outputs. This is where the dict index comes in handy:
for conn in connectorData:
f,t = conn
fromNode, fromConn = index[f]
toNode, toConn = index[t]
fromConn = fromNode.getOutputByName(fromConn)
toConn = toNode.getInputByName(toConn)
fromConn.connect(toConn)
which is to say:
for each connection pair in connectorData:
f,t = connection pair
fromNode, fromConn = the node reference and label stored at the location "f" (an output id) of the dict "index"
toNode, toConn = the node reference and label stored at the location "t" (an input id) of the dict "index"
fromConn = the NEW output id of the output in question, which we get from the node using getOutputByName("label name")
toConn = the NEW input id of the input in question, which we get from node.getInputByName("label name")
connect the output to the input
Player
The file App/player.py contains all of the code necessary to run an ARIEL sketch. You may notice that the class names in player.py are identical to those in node.py; this is part of the compiler mechanism that translates a sketch into a working piece of code.
For every node.Node class, there is a player.Node class. Whereas node.Node represents the user interface for the environment that allows you to do the actual programming, player.Node is a representation of the functionality of the Node.
When player.py is run, another openFrameworks app is built and instantiated. The ArielApp at the bottom of player.py is the framework for the app: it contains the setup, update, and draw functions, as well as the event callbacks. By the time the script reaches ArielApp.setup(), however, most of the work has been done by the compiler (documented below).
Each player.Node contains compute(), update(), and draw() functions. compute() is where the work is done; update() tells the node's outputs to push their new values to whatever inputs they are connected to; and draw() is exposed for nodes that draw to the screen.
When an app is exported, the order of the nodes is preserved. This order is determined by when each node was created; the first node created in the environment will be the first in the list (unless Order Edit Mode has been used). The update and draw loops in player.ArielApp reflect this order:
def update(self):
for n in self.nodes:
n.compute()
n.update()
def draw(self):
...
for n in self.nodes:
n.draw()
There can be some weirdness if nodes further down the chain get updated before their 'parents' do; unfortunately, this is often the case. It often presents itself as 'lag', where images or numbers take longer than normal to respond to interaction. When doing time-sensitive calculations with numbers that involve many nodes, this problem can seriously affect performance. Order Edit Mode was designed to deal with this issue.
To see more on how nodes are structured, look at the Creating New Nodes page.
Compiler
The compiler's job is to take an ARIEL sketch that has been cPickled (using the builder.py/ArielBuilder.loadGraph() function discussed earlier) and turn it into a functional application.
player.py must be run from the command line with a filename as an argument, i.e. python/bin/python player.py scene.ariel. When hitting the 'Run' button from ARIEL Builder, builder.exportAndRunGraph() saves the current sketch as scene.ariel and tells Terminal to run player.py with the argument scene.ariel:
def runGraph(self,fname):
args = ""
if self.fullscreen.currentChoice == 1:
args += " fullscreen"
if os.system(sys.executable+" player.py "+fname+args):
gui.doPopupMessage("Scene exited abnormally.","Check the Terminal window for additional error information.")
Calling this terminal command within builder.py results in a blocking operation, i.e. running the sketch will prevent anything from taking place in the builder environment until the sketch is either closed or fails. This saves a ton of processing power for use running the sketch but has the unfortunate side effect of blocking the user from interacting with the environment– a source of confusion for some.
Upon running player.py with a filename argument, the file is de-pickled into two arrays: nodes and connections. Skip past the concrete class declarations to the line import node, where the compiler gets started:
import node
for obj in dir(node):
c = getattr(node, obj)
try:
if issubclass(c,node.NodeInput):
exec "arielplugin."+c.__name__+ " = NodeInput"
if issubclass(c,node.NodeOutput):
exec "arielplugin."+c.__name__+ " = NodeOutput"
except:
pass
for f in os.listdir("User Nodes"):
if f[-3:] != ".py":
continue
sys.path.append("User Nodes")
module = f[:-3]
exec "import "+module
exec "mod = "+module
if not eval("hasattr("+module+", 'arielplugin')"):
continue
for o in dir(mod):
c = getattr(mod,o)
if hasattr(c,"ARIELCLASS"):
exec o+" = c"
else:
pass
This section goes through each NodeInput and NodeOutput subclass, adds it to a module called arielplugin from arielplugin.py, and stores it so that User Nodes can translate their special input and output types to the player.NodeInput and player.NodeOutput types. This piece of code is only useful when User Nodes are implemented in an ARIEL sketch.
On to the nodes themselves:
for n in nodes:
classname = string.splitfields(string.split(n[0])[1],".")[1]
newNode = eval(classname + "()")
...
For each node in our list, get the classname (without the node. prefix) and use eval to instantiate a node with it. This is the magic: when eval(classname + "()") is run, the class being instantiated is a player.Node as opposed to a node.Node class, making a translation between player.py and node.py. This is why the classnames must be the same across player.py and node.py.
...
inputs = n[1]
outputs = n[2]
params = n[3]
for i in inputs:
label, number = i
c = newNode.getInput(label)
c.id = number
for o in outputs:
label, number = o
c = newNode.getOutput(label)
c.id = number
for key in params.keys():
cmd = "newNode."+key+ " = " + `params[key]`
exec(cmd)
activeNodes.append(newNode)
Since the nodes have already been instantiated, the NodeInput and NodeOutput objects within Node.__init__() have already been created. Getting a reference to them is a matter of calling newNode.getInput(label), which allows this operation to give them an id. This id will be used to create the connections (much like in node.py/ArielBuilder.loadGraph()).
The final section turns node.Node parameters into player.Node class variables. For each parameter found in the params dict the key is turned into a variable using cmd = "newNode." + key + " = " + `params[key]` . This means that if a node.Node parameter for "number" is saved (using Node.setParameterDict()) as d["number"], it will show up as a variable in the player version of the node, e.g. player.Node.number.
Finally, each node is appended to activeNodes, which is the node list that will be given to the ArielApp class.
A simple operation turns the connection information from the original file into actual connections between nodes:
for c in connections:
f = None
t = None
for a in activeNodes:
f = a.getOutputByID(c[0])
if f:
break
for a in activeNodes:
t = a.getInputByID(c[1])
if t:
break
f.connection = t
Here the id from each connection is used to get a reference to the newly created NodeInput and NodeOutput. The output stores a reference to the input, creating the connection necessary to pass information along.
Now the activeNodes list contains all of the Node instances with their connections mapped. These become self.nodes in ArielApp.setup(), at which point several for loops check for special cases that might require additional linking.
These special cases include the 'variable' system in Builder: the nodes 'get var' and 'set var' allow you to store a number without using cords. In order to implement this, the nodes store special references to each other that must be established during setup. ArielApp.routeSettersToGetters takes care of this.
At the time of this writing, the only other objects that require special setup in ArielApp are player.MouseInput and player.GUIButton, which are passed mouse events within ArielApp's mouse event handlers, and player.keyEvent, which must be passed the key events from ArielApp.keyPressed.
Advanced Topics
Order Edit Mode
Order Edit Mode, available in the Edit menu of ARIEL Builder, was designed to deal with the order issue that arose from the way ARIEL Builder stores its list of nodes. When nodes are created, they are added to the list of objects maintained by gui.PSGObjectManager. The first node created is the first node in the list, etc. However, users rarely create nodes in the top-down order that they will be arranged in once they are connected to each other. Since player.py runs the sketch using this order, it can cause lag and, in serious cases, faulty data and calculations.
When a user enters Order Edit Mode, their sketch 'freezes' and an orange circle appears next to each node with a number inside it. The number corresponds to the node's spot in the list starting with zero. If the user clicks a circle, types in a new number, and hits return, the node will now occupy that spot in the list.
If a node was in position #19 and the user types in '10' and hits return, that node will become #10 and the nodes that were formerly 10-18 will shift to become 11-19. Users are recommended to start from 0 to avoid unintentionally shifting groups of nodes around.
This mode is powered by the builder.TagController class. Each time the 'Order Edit Mode' menu item is selected, the function builder.ArielBuiler.showOrdering() is executed, which creates a new TagController in the variable ArielBuilder.tagController.
Mouse and key events are forwarded to the tag controller in this rather chunky manner:
def mouseMoved(self, *args):
if self.tagController == None:
apply(gui.manager.handleMouse,args)
apply(gui.manager.palletScrollBar.mouseMoved,args)
else:
apply(self.tagController.mouseMoved, args)
The switch in placement is performed using TagController.requestSwap(old_pos, new_pos).
Containers
The container system is an attempt to provide a method for users to create new nodes out of collections of existing nodes.
A Container is a node that points to an ARIEL sketch file. When the file is loaded into the node, it is searched for special nodes called ContainerInlet, ContainerOutlet, and ContainerTitle. These determine how many inputs and outputs the Container has.
To create a Container, the user must create a separate ARIEL sketch that contains the desired functionality. ContainerInlet and ContainerOutlet nodes represent the inputs and outputs.
The container system works using the same process as Loading Sketch Data. To create the Container node in the environment, node.Container.parseFile() unwraps a cPickled file (given as a parameter) and calls node.Container.parseExistingData() to look for inlets, outlets, and a title.
At this point the container object is still only a representation of the file; the actual reading and 'unraveling' of the file doesn't happen until the user runs the sketch.
Within builder.ArielBuilder.exportAndRunGraph() there is a statement that checks whether or not there are any Containers in the sketch. If there are, it runs ArielBuilder.stitchContainers(), which in turn runs ArielBuilder.nodesFromContainer(container_data) for each container found. This function closely resembles ArielBuilder.loadGraph() in the way that it creates new nodes, however additional operations are necessary to stitch the nodes together.
The stitching process works by taking the outputs that were connected to the inputs of the Container object and attaching them instead to the nodes that the ContainerInlet objects were connected to. Likewise, any Container outputs are removed and the actual node outputs are used that connect to the ContainerOutlet objects within the container sketch.
There is a known issue with using multiple outlets; the stitching process can sometimes fail to build the connections correctly if multiple ContainerOutlet nodes are used in the container sketch.
This process does not currently jive well with the Order Edit Mode system.
Recommendations for a New Version
...