SW Architecture

A very good [article](SW Architecture A very good article by Paul Rathgeb explains the SW structure of WeIO system.

Tornado Server

Tornado is a web server written in Python. Originally used for Facebook wall, open sourced.

Two things that we needed from Tornado:

• Asynchronous execution - i.e. handler is called upon epool notices filedescriptor change
• WebSockets - i.e. "server push"

We use asynchronous execution in many cases, but good example is when a browser client sends a request over some route. There is a special route called /api via which browser sends commands like analogRead() or digitalWrite() to the Python backend which executes these commands on the board.

Tornado lets us practically observe any file descriptor and act upon the change.

We use WebSockets for “server push”, i.e. to send the data from the WeIO board (server) to the browser client. Let's take example that we observe a temperature in browser client on the mobile phone. Physical events happen in the nature asynchronously and when WeIO board detects the change it must be capable to “push” this information towards browser on mobile phone. This can not be accomplished using classical approach where only client can send HTTP requests demanding data from the server. That's why we use WebSockets, which stay opened during the whole session and allow bidirectional data flow between server and the client. So this way WeIO board is capable to push new information about the temperature to the mobile phone as soon as the event happens in the real world.

This approach is also very useful for development, as we are using WebSockets to push console line from the Linux to the IDE in browser - this way we can have print-outs directly in “Console” part of the IDE, which is essential for debug.

Tornado Documentation and Examples

Here is a lot of documentation: http://www.tornadoweb.org/en/stable. Here is a very nice tutorial: http://developer.mbed.org/cookbook/Websockets-Server

For starting with Tornado I suggest you that you clone my repo: https://github.com/drasko/tornado_boilerplate.git and look at example how it is used.

WeIO uses 2 Tornado Instances

WeIO application is consisted of two parts:

• IDE
• User program launcher To keep these parts decoupled each is run in the separate Tornado instance over 2 different ports (port number is configurable in config.weio JSON configuration file).

There are several important reasons to keep these parts decoupled:

• Avoid to block IOLoop of IDE with user program - user can write any program, and potentially have a infinite loop or block on some resource forever. If it was executed in the IOLoop of the IDE Tornado, it would block it and freeze the IDE, after which user would not even be capable to stop and debug the program
• Potentially remove IDE Tornado when going to production

These two instances are main part of the WeIO app and are located mostly in files:

weioServer.py - for IDE Tonado weioRunner.py - for User Tornado

How it Works

Here is how this mechanism works:

• IDE Tornado starts and serves WeIO IDE
• It launches User Tornado as a Python subrocess via weioPlayer.py
• User Tornado blocks on the pipe filedecriptor waiting from the “START” command issued from the user by button “PLAY” in the IDE
• User uses IDE in the browser to write it's programs (equivlent to Arduino Sketch-es)
• Once command “START” is recieved, User Tornado launches all user programs in parallel

Tornado Quirks

In order to present simplest possible interface to the user, WeIO missuses Tornado (on purpose - we know what we are doing).

Tornado is supposed to be single-threaded app using epoll for ioloop to loop through the events and call adequate non-blocking handler for each event. Non-blocking is a key word here! Programmer must avoid to block the ioloop.

Since here we are just giving the a possibility for users to write the programs, we can not have any idea if they will write a blocking calls or make infinite loops (and probability that they will is very high). That's why we must launch user programs in separate threads, so that if they block, they do not block whole ioloop.

This however introduces another difficulty - you can not kill the thread in Python, and we do not know will the user exit thread nicely (and he probably will not). So threading User Tornado will demand killing whole User Tornado and restarting it again on every “PLAY” or “STOP”. However starting Tornado server on emebedded machine takes too much time and is not acceptable.

Finally, trick that we use is that we:

Launch special Python multiprocessing “launcher” program on PLAY, that is launched from User Tornado That launcher process threads into user programs On STOP we can kill just this launcher process and all unexited user threads will die This way we avoid to restart User Tornado every time on STOP/PLAY.

N.B. Forcefully killing threads is dangerous business in Python - especially if some thread is holding some resources, like opened file - it might be corrupted.

Shared Variables

Since all user programs are threads of a process “launcher” they can share global variables, provided synchronization mechanisms.

However, events that handle JS API and custom events are executed in the context of User Tornado, which is different then the context of launcher program (because launcher is started from the User Tornado using Python multiprocessing module - i.e. as a separate program).

They can however still share variables, but only using Python multiprocessing defined shared variables (https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes)

Bojour ZeroConf Discovery

When WeIO is in STA mode, i.e. when it is connected to the home WiFi router, it will obtain some IP address form the DHCP. Since there is no displays on WeIO we will not know which address it is. So we will not be able anymore to point our browser to WeIO board.

WeIO uses simple discovery mechanism called Bonjour or ZeroConf (http://en.wikipedia.org/wiki/Zero-configuration_networking). This is a service which Apple open-sourced and is often used to find and reach printers on the network.

Every WeIO board is given it's name (Linux hostname) during the initial configuration. Using this name we can reach this board in LAN by simply accessing to “myBoard.local”. Suffix .local is necessary by protocol rules.