Tortoise - NetLogo/NetLogo GitHub Wiki
(summary of development activity as of April 2013; other wiki pages may have more current information)
The Northwestern team's development efforts are focused on producing a version of NetLogo that will backwards-compatibly run existing NetLogo models in-browser on a JavaScript + HTML5 base.
This new version will replace the NetLogo applet, which was our old way of running simulations in the browser. Our applet ran on the JVM (Java Virtual Machine). But JVM applets don't work on mobile devices and are a dying technology even on the desktop, due to security problems, slow load times, and other issues. They are no longer suitable for use in schools.
By contrast, JavaScript and HTML5 are secure, nearly universally available, and (in the latest browsers) fast enough to compete with the JVM.
Our top priorities for this work are:
- backwards compatibility with the existing NetLogo language and existing NetLogo models
- embeddability, including integration with Concord Consortium's Lab environment
Uri Wilensky is overseeing the Northwestern team and directing the design. Seth Tisue is the technical lead and is doing the main work on the new compiler and agent engine code, with some open-source participation from Josh Cough. Jason Bertsche and Bryan Head are building the server-side web app, the browser front end (the latter based on an initial prototype by Philip Woods), and the Canvas-based renderer. Nicolas Payette is responsible for the event-sourcing code in server-side NetLogo. Our main technical liaison at Concord Consortium is Stephen Bannasch.
The first phase of work involved evaluating and selecting technologies.
Like NetLogo (since 2011), our new work is open source software, under the same license, the GNU GPL (GNU General Public License). Source code, plans, and design documents are in public repositories hosted on GitHub. Development work is discussed in the open on a public mailing list, netlogo-devel, hosted on Google Groups.
Although the new system is JavaScript-based, we are not abandoning the JVM-based NetLogo, which will continue to exist both on the desktop and server-side. Server-side code will support the browser-based system. Nearly all of our JVM-based work is now done in Scala, not Java. Scala compiles to JVM byte code. We upgraded this year to JVM version 6 and Scala version 2.10. Our build system is the Scala-based SBT 0.12. (This year we completed our transition from Make to SBT.) We run continuous-integration tests on GitHub Actions.
For targeting JavaScript, our programming language of choice is CoffeeScript. We researched other so-called "alt-JS" languages that compile to JavaScript, especially hoping for a statically typed alternative, but nothing seemed both lightweight and mature enough to bet on. (Alternatives we researched included Fay, Roy, Elm, Typescript, ASM.js, and ClojureScript.) SBT easily incorporates CoffeeScript code into our projects via the coffeescripted-sbt plugin.
Our web framework of choice for server-based applications is Play 2. We're using WebSockets for client/server communication; not every browser in the field currently supports them but we expect that will change soon. Using WebSockets has somewhat restricted our choice of virtual server hosting; we tried Heroku and dotCloud but they don't currently offer WebSockets and the workarounds they offer are cumbersome. Currently we are hosting our web apps on Linode.
For 2D rendering in-browser we are using Canvas. We have conducted some performance experiments with SVG+D3 instead, but with disappointing results; still, this is a decision we may revisit in the future.
We are targeting modern browsers only, on the assumption that in just a couple years today's cutting edge browsers will be in widespread use.
For testing purposes during development, we're using Rhino to run JavaScript code on the JVM. Once our automated tests pass on Rhino, we manually test in the browser using Firefox, Chrome, and Safari. (We don't yet have any automated browser-based testing.)
We may eventually replace Rhino with Nashorn. Nashorn is new and fast-evolving; it should be much faster than Rhino and support newer versions of JavaScript.
We have built several prototype systems to evaluate technology choices and explore possible designs and architectures.
The first prototype of what would eventually become Teletortoise (see below) was Wolf. It used the Play framework and WebSockets to run a headless NetLogo instance on the server, accept commands from a browser-based command center, and display the results by rendering PNG images on the server. No actual code from Wolf survived into later versions, but the project helped cement our confidence in Play and WebSockets.
Wolf's successor was a NetLogo web app built using Node.js and Play. We never planned to keep Node.js in the long run, but using it temporarily allowed our intern to work entirely in JavaScript for both client and server code, without having to learn Scala. This app had a richer user interface than Wolf, and rather than using PNG images, it rendered simulation state using JSON updates. Some of the code has survived for reuse in our current-generation web apps.
Bryan Head re-implemented some well-known NetLogo models directly in CoffeeScript, in a prototype project he called WebLogo. This was our first experience with CoffeeScript and we found that we liked the language and that the performance was surprisingly good.
Although we didn't build it ourselves, we've been closely monitoring Redfish Group's AgentScript project, which (like WebLogo) involves writing NetLogo-like models in CoffeeScript. We expect to learn a lot from their experiences and possibly share code in the future.
To pilot integration of NetLogo with Data Games (a data graphing and analysis environment), we built a JSON data exporter for BehaviorSpace that output NetLogo parameter-sweeping experiment results in a format that was readable by Data Games to visualize and explore the data in both graph and table form. This worked well and will serve as a basis for future, closer integration between NetLogo and Data Games.
Although we aim to jettison applets in the long run, for prototyping we continue to support the applet version of NetLogo, which with our assistance was integrated by Concord Consortium into their Lab environment. In this applet-based prototype, data from a running NetLogo simulation is displayed live in both graph and table form in Data Games.
We are simultaneously pursuing two approaches for running NetLogo simulations in the browser. Our code names for these two approaches while they are still under development are “Tortoise” and “Teletortoise”.
Tortoise is the all-JavaScript, entirely client-side system described above, in which the simulation runs entirely in the browser, using JavaScript for computation and HTML5 for display.
Teletortoise is a hybrid client/server system where the simulation runs on the server using the JVM version of NetLogo, using the browser only to render the results (including live animation of simulation progress).
| Tortoise | Teletortoise | |
|---|---|---|
| compiler | server | server |
| engine | browser | server |
| graphics | browser | browser |
Both Tortoise and Teletortoise are important to NetLogo's future. Teletortoise isn't just a prototype of the eventual Tortoise system, but will continue to exist on its own. There are pros and cons to both approaches:
- Tortoise requires little or no server-based support and is not limited by bandwidth.
- On the other hand, Teletortoise supports the full feature set of JVM-based NetLogo now, features that will take years to duplicate in Tortoise. It also supports existing JVM-based extensions.
- Teletortoise will serve as the basis for our next-generation HubNet technology for participatory simulations.
Tortoise is a single-user system, since the simulation is running in the user's own browser, but Teletortoise is a multi-user system, where any number of participants can observe and control the same model run together and discuss it in the integrated chat room.
Tortoise and Teletortoise are not entirely separate projects. Much of the code will be shared in common. In particular, the vast majority of the JavaScript+HTML5 code for the user interface in the browser is the same in both, and much of the server-side code for managing user sessions is also shared between them.
This is possible because both Tortoise and the JVM-based NetLogo have an “event-sourced” architecture, where running a simulation generates a stream of state-change events, which can then be received and rendered separately. In Tortoise, the events are both generated and handled by the browser. In Teletortoise, the events come from the server.
Since server-side NetLogo and Tortoise both generate the same events, the same browser front end can be used for display and interaction with either back end. Regardless of where the events are coming from, they can be handled uniformly.
Events can also be recorded and replayed later, so this architecture can be used to play back previously recorded model runs. Event types include agent state changes, drawing events (pen marks and stamps), and plotting events. (This year, we substantially rewrote NetLogo's plotting and drawing subsystems so they generate these new events.)
This event-sourced architecture was originally developed for model runs recording and replay for our ModelSim project, but we had in mind all along that we would also use it for Tortoise, Teletortoise, and HubNet.
The Tortoise compiler is written in Scala 2.11 and reuses the early phases of the existing NetLogo compiler. This year we substantially reworked several major components of the compiler in order to make them usable by Tortoise, most significantly, a full rewrite of the first phase of the NetLogo parser. The various phases of compilation were refactored and divided into a “front end” and “back end”, with the front end consisting of only the phases used by Tortoise.
For now the compiler generates JavaScript directly. Later, we may change to generating CoffeeScript instead, then compiling that to JavaScript. For now, generating JavaScript directly simplifies our development process.
Since the compiler only runs on the JVM, compilation can't happen in the browser. It must happen ahead of time, or if the on-the-fly compilation is needed, it must happen server-side. (In the long run, it might become possible to compile Scala to JavaScript, in which case compilation could happen in the browser again. And/or, we might also eventually build a desktop app that bundles a JavaScript-based client with a JVM-based “server” that actually runs locally behind the scenes.)
The Tortoise compiler currently supports the following subset of the language:
- complete NetLogo syntax (since the real parser is used)
- literals (numbers, strings, booleans, and lists)
- procedure definitions (
toandto-report) -
ask,while,if,ifelse,repeat -
set,let -
globals,turtles-own,patches-own -
agentsets(limited functionality) clear-all-
and,or -
forward,left,rt,setxy,die -
create-turtles,create-ordered-turtles,sprout selfwith- arithmetic operators:
+ – * / = != < > <= >= output-print
Not all of these features are implemented in their full generality, but basic uses are supported.
Besides the compiler and the user interface, the other remaining major components of Tortoise are the event coalescer, the engine, and the renderer.
The engine represents turtles, patches, links, agentsets, and other NetLogo entities as data structures which are accessed and manipulated by the JavaScript code generated by the compiler. This code handles things like agent locations and movement, distances and angles, agent variables, agent creation and destruction, agentset operations, world topologies, and so forth. So far we have only replicated a small part of NetLogo's agent engine in CoffeeScript. A major challenge in the coming years will be to greatly expand this subset without doubling our overall development, maintenance, and testing effort. We are investigating technologies that may be able to assist with, such as automated translation of Scala code to JavaScript.
The update coalescer receives a stream of events from the engine (the Tortoise engine in the browser, or the Teletortoise engine running on the server) and merges them as they come in, resulting in a representation of the current state of the system as input to the renderer.
The renderer draws the current state of the turtles and patches in an HTML5 Canvas.
All of these components are written in CoffeeScript.
Our first major goal was to build a complete, working system as soon as possible, end to end, with a rough minimal version of every component in place, and a basic suite of automated tests passing. We successfully achieved this goal, with the first version of Tortoise going online in February, and initial integration of Teletortoise with the Lab environment achieved shortly thereafter.
The models we have working so far in Tortoise are Life and Termites. (We plan to tackle Climate Change and Wolf-Sheep Predation next, but to implement these, we'll need to support turtle breeds. Turtle breeds are just one of many NetLogo language features that the compiler doesn't support yet.)
We are using Rhino to run the JavaScript output of the compiler on the JVM using the same RNG as the regular JVM NetLogo, and then we compare the result JSON event stream to make sure they are identical. This ensures that Tortoise and NetLogo behave identically (on identical random number streams).
Even without any significant performance tuning work having been done, Tortoise is already only about 10x-15x slower than the JVM NetLogo. We expect to close much of that gap in the future. (Note this figure is engine performance only; we haven't benchmarked graphics rendering performance yet.)
This first version of Tortoise works on both desktop browsers and iOS-based mobile devices (iPhones and iPads). It doesn't currently work on Android devices, but we plan to address that.
- Teletortoise:
- Tortoise:
- Teletortoise embedded in Lab:
