This is a development log for Orbital that is being kept in a wiki. Generally it is being organized in reverse chronological order - most recent thinking is first. Note that the hard work of this project is simply the thinking itself. There is a fair amount of repetition and reframing. See the website at https://orbital.foundation for the bigger picture.

Influenced by burgeoning climate issues we've decided to focus on a single demo to explore coral reef ecosystems. This is ongoing.

While the work was unfunded this year we did have some clarity on fundamental architectural decisions, and the framework has been formalized to reflect those ideas:

• SDL (System or Scenario Definition Language). The new https://github.com/apple/pkl configuration language shows the value of configuration grammars that allow some code - this is validation for our configuration approach which simply uses Javascript. Although we encourage a declarative pattern we felt it was important to allow procedural code. This youtube video also echos similar sentiments: https://youtu.be/lAxXWYAIt4k?si=WsO7TpF3NmjzVBPD . In our case we are staying with our own grammar which is better established than pkl.

• ECS (Entity Component Systems). Agents were typically described declaratively as a collection of components such as a transform capability and a geometry capability being expressed as simply { transform: { xyz:[0,0,0] }, geometry: "/assets/manta.glb" }. We now see ECS more in terms of database driven reactions to state changes rather than formalizing ideas of components or systems. In Orbital we have 'observers' that can react to state changes on the database to produce component or system like effects. A 'tick' that is sent with the screen refresh can be caught by an observer and used to implement an ECS system.

• Reactivity. The most clarity for us was moving to a more clearly reactive system. React, while complex from a novice perspective, does formalize a horizontally scaleable pattern that allows for much more complex code bases. We use an event architecture where events flow through the system and observers can react to those events. Typically most events are database writes; and often are writes to specific component properties of specific entities.

This year we focused on defining "what is an app?" within the context of a future oriented AR web browser.

It turns out an app is a whole pile of libraries and tools and security policies - and that there are tensions between static or early binding of apps and late binding. We see a shift towards late binding, where apps themselves are just small bits of scripting glue that lash together a pile of pre-existing components. In this sense Orbital can be seen as glue for orchestrating other pieces of code. There are strong similarities between cargo crates, package builders, npm, webpack, and cloud rust containers (such as cloudflare agents) - and what we think of as apps. Overall - defining an app more formally lets us run apps together better.

We see that an AR browser needs a well defined "app runner" / hypervisor or thread management scheme with formal ways to dynamically load and unload libraries and resources. It needs to be able to run multiple applications at once - and for those apps to communicate with each other. Apps and agents also overlap conceptually, where an app is a pile of orchestrated agents. This also dovetails with ideas around agent models and agent sandboxes. We see a difference here from browsers today - which run one app per page, and apps don't really talk to each other. Browsers in the future run all apps on the same "page" in augmented reality. They really are no longer "apps" but are more like "applets" or even individual agents or "little digital friends" that wander around in your field of view, decorating your field of view with useful information. This also implies a lot of policing of what they can and cannot do - or else you end up with a poor user experience (which Keiichi painted so well in Hyper Reality -> https://www.youtube.com/watch?v=YJg02ivYzSs ).

We recognize a need to more formally define an "app container" or "agent server" and that a browsers today are effectively a kind of app container, and so is a desktop, and so is an iOS enabled device, and so is a cloud compute service. All around the world we see that there are containerized silos that run piles of apps - although often those apps do not leverage each other. We each have a personal server that runs our own apps (our phone) but we also talk to apps in the cloud - running in cloud containers. App containers do require well defined apps, and app containers are effectively a core part of a future browser as well. App Containers also similar to agent model simulations - a container that runs a bunch of interacting blobs of code.

In 2021 we were focused on a performant thread-manager and dynamic WASM based app loader written in Rust. Rust is too slow to develop in and developing in Rust before ideas are mature is a form of premature optimization. Instead this year we presume that a performant future oriented browser already exists and is capable of running whatever we tell it to. And instead we focus on the highest level description of what an app "is". This lets us focus on describing apps at a high level, and on ideas like distributing apps over the net, and persistent and durable apps that run on device in the background forever (unlike web apps). We found that more formal definitions of security and capabilities is important to have durable persistent apps that can flit about the net, be loaded and unloaded, move computation as needed, and be trusted to do so. And we also find that we want to have a more lego like grammar to let novice users describe complex scenarios. In this ongoing work we are finding ourselves focused almost entirely on "SDLs" or scenario definition languages that capture a highest level manifest or expression of an app.

These are the general classes of usage we envision - from simplest to most complex (circa December 2022):

1. Interactive Communication -> the creation of tiny simulations visualized in 3d, 3d stories and vignettes and the sharing of them between friends. While these don't need to be 3d or visual at all, we see a role for lightweight communication using behaviors not just images or text. Visualization is often interleaved with execution - but we do think visualization is secondary in importance, and behavior is more primary. Effectively a user should be able to make a small representation of an issue, and share that issue - proxying themselves in a sense.

2. Sense Making for Individuals -> We see that individuals will also have a passive use for Orbital, to run a personal server that can run a bunch of agents all the time that are looking out for that persons best interests. They can advise when friends are nearby, or give advice about activities to do, or even just show data like if an area is dangerous or not.

3. Sense Making for Groups -> At scale it become possible to run a bunch of agents to simulate harder problems, such as helping model civic outcomes of changes to law - like you might want to go to city council and argue that they should block roads in your neighborhood to reduce traffic, and you can present a simulation of that by modifying the traffic rules in your neighborhood in a simulation.

4. Digital Twins -> An MMO is effectively pile of agents running on a server in the cloud with rich visualization. I do see a use case to help people create their own dedicated worlds - similar say to https://www.palia.com/ which should be out next year - but for me my interest is more focused on digital twins and real world problems at scale.

The milestone 5 statement of work is "Stage public website and arrange materials to showcase SDL and the white paper. Publish the white paper to github under the MIT/Apache open source license."

What we're doing now is focusing on language, phrasing and a single demo.

Currently the phrasing is that "Orbital is an app container for running rich apps."

We see Orbital as similar to an operating system microkernel in that it manages the loading, unloading, thread management and messaging between applications and application services. It is also similar to a package manager such as cargo crates or npm in that it joins together pieces of an application on demand.

The value we perceive is that Orbital exists to expand on design choices relative to application containers today. We believe that apps in the future will be built out of small pieces loosely joined and that computation will be distributed over the Internet rather than necessarily always on device. We also believe in reducing barriers to creating and publishing apps.

Orbital has an emphasis on self sovereign identity, granular security, shared state, persistence, distributed computation and 3d user interfaces. We introduce a 'scenario definition language' where applications are wired together out of dynamically loaded services such as computer vision modules, deep learning modules and rendering and visualization modules.

Orbital apps are carefully scoped to currently run within a web container, this allows us to prototype much more quickly. The intent is to ultimately run natively on dedicated hardware such as next generation augmented reality glasses. We currently (and will continue to) support javascript as the best scripting language of choice but will later include WASM support. We currently target Babylon3D as a 3d rasterizer but will target Bevy3D for embedded environments.

This milestone is focused on cleanup and documentation. This includes:

• Publish Source
• Web Sandbox -> provide a web sandbox for third parties to try play with the engine
• Promote / Publish -> Stage public website and arrange materials to showcase SDL and the white paper. Publish a white paper to github under the MIT/Apache open source license.
• Speak -> Speak and build support
• Community -> Continue to see if we can foster an active community
• Feedback - begin to collect feedback from third parties and start to include other participants
• Document -> provide reference docs of precisely how an ordinary developer can use Orbital to make and distribute rich apps

The original SOW for this deliverable is "Target native renderer (bevy3d) as a platform independent SDL execution environment. Bind the SDL back into the “Orbital” browser project."

However a vehicle simulation is such a good example of a use case that we will continue to focus entirely on a digital twin based around an urban setting with vehicles as a demo. So we will pivot this deliverable to focus on a car simulation rather than Bevy3D.

A car simulation demo exercises several key ideas. First it is a well known topic, secondly it shows heterogenous agents running in a sandbox. Third it also dovetails with larger goals for a personal server. It also is visual - and importantly it is simpler to write than say a reef ecosystem simulation or a virtual personal desktop with many helper agents. Also, from CARLA we see ideas around separating computation from rendering - and I want to explore those ideas.

Notes:

20221120 M2 Orbital Ongoing Goals Form Versus Function

Our stated goals for this milestone were to "Build demos, exercises and examples to showcase the value of an SDL. This will consist of demos such as a web sandbox augmented reality SDL based scenario, and also demonstrate and exercise a cloud compute “digital twin” sandbox for running multiple scenarios at the same time in a multi-participant scenario."

Our vision has evolved since year 1. The original vision for Orbital was a "blue sky browser" and we quickly saw a need for heterogenous agents to run on a personal server - effectively to do sense-making for the user. We had then focused specifically on how those agents are described ( the SDL Grammar ). Now I think we really need to provide examples of compute sandboxes that agents can run in. From the last milestone we have several rough examples (a digital twin project, a virtual desktop) but these are only rough.

Continuing down an App Container path, now we want to better identify actual use cases and potential users. We have a running system that lets users load and run apps, and that roughly follows some of our ideas around agent pools. It still needs work, but we're starting to get to the level of examples. Current examples (which are rough) are:

• An AR Desktop -> exploring ideas around what a user interface could be like
• A multiplayer sandbox -> simply letting multiple people create objects and see each other
• Computer Vision Demo -> testing out routing and data movement between different services and our SDL grammar
• Virtual Venice -> a rough idea to explore building a digital twin of Venice
• Cloud Reef -> an idea to explore a reef ecosystem simulation
• Zone Picker -> an idea for navigating geospatially anchored areas
• Carla Clone -> driving Carla remotely? (Or I think now we will actually focus on cloning some of the Carla functionality instead)

I also do want to now start looking at a more mature model for representing dynamic actors and state as well - in this case I will use an ECS model such as https://blog.mozvr.com/introducing-ecsy/ .

Notes:

• Orbital Services Principles
• Orbital Filesystem Persistence and Networking
• Orbital Examples
• https://orbital.foundation

The stated goal of this milestone is to target the web first to quickly bind a fairly comprehensive set of SDL nodes that can express concepts at three levels:

1. Wiring together high level “apps” out of behavior components (video inputs, tensorflow / opencv modules, rendering output).
2. Demonstrate portable description of typical 2d feature rich layouts; fonts, 2d rendering primitives, 2d animations, 2d effects and user input (buttons, input boxes, draggable zones).
3. Demonstrate portable description of 3d layouts: lights, cameras, objects, behaviors.

Internally the architecture has been migrated to javascript from Rust. We have the same concepts but now in a lighter language that lets us iterate more quickly. There is a concept of a "pool" that runs services, either locally or in the cloud. We have a "view service" that supports much richer capabilities than our previous Rust example. We have multiplayer networking and persistence. We also put up a website - and now can run Orbital inside of a web container. This lets us more easily share our thinking. We will also let users make and share their own example scenarios.

We did look closely at Carla and liked some of the thinking there quite a bit. Carla can be driven remotely through scripting, effectively it acts like an app container for a a domain specific use case. There is a message passing interface to the core engine. However that engine itself is bound to a rasterizer/view framework. Orbital can be seen as a slightly more general version of this, it runs arbitrary agents, although there can be domain specific pools, and Orbital is not necessarily focused on rendering (it has no renderer at the core).

Notes:

• What Is Orbital

The stated goal of our second year first milestone is "Produce high level road map describing open source SDL approach. Capture and document all the nodes that we plan to support in the SDL. Demonstrate early web bindings."

Given market trends towards 3d graphics and heads up displays, we are also beginning to see more applications, use cases and kinds of customers. One kind of customer we find especially exciting is people who want to build or use digital twins. We see multiple well funded digital twin initiatives emerging - from work by Meta to a Digital Malta funded by the Maltese Govt to others. We also see personal AR overlay type use-cases as computationally similar to digital twins, and we see digital twins providing real value to both consumers and industry.

At the same time I see a few digital twin builders are fixated on appearances - graphics and rendering - rather than computational foundations. I am honestly surprised by the enduring popularity of the legacy brute force development pattern where an application must be torn down and restarted whenever any agent within it changes. For even a small change to 0.01% of an app - the entire app must be rebuilt. Unreal is an especially tragic example of this anti-pattern. Any change to a client application requires the client to be recompiled, re-distributed to all users, all users must stop and restart the app, the servers to be recompiled and rebuilt. Effectively the entire state of the system to be suspended or torn down, and then rebuilt and reloaded and rerun. It's inefficient at best. Beyond this the Unreal marketing pitch tends to focus heavily on visualization, and tends to encourage baking in art assets into the builds themselves rather than dynamically or late loading assets - therefore even changes to art assets force rebuilds of apps - and apps are burdened down by art assets which further slow iteration. This kind of rudderless technical leadership simply isn't going to deliver the future and it is a pattern to avoid.

I now tend to see Orbital as an "app container" - something that ordinary consumers will run on embeddable devices, especially heads up displays - and as well in the cloud. Users want to be able to load and run rich applications, have those applications be trustworthy, and for it to be easy to write and share their own small apps or services. People should be able to create proxies that reflect their own will, that can allow others to explore an idea, effectively as a form of advocate or even a shared experience.

We see that an app container is a way of thinking about computation - operating systems, web browsers, cloud compute. App containers can be run on personal servers (such as on a phone or AR display), or in a cloud compute situation such as a massively multiplayer online simulation, and that apps (or agents) can migrate back and forth, or even have parts of their computation migrate back and forth between device and cloud as best needed for various scenarios. An agent that needs to interrogate a large medical database may as well move to the data and be run there, rather than moving the data to the client device.

We also feel that users should be able to write their own "apps" or "agents" that they can drop into an app container, and that agent can interoperate with other agents without having to tear down and restart the entire system.

Within the app container framing we still feel the critical piece is to continue to define how to easily express apps as a whole. In that light we continue down the SDL path with these goals:

1. Continue to make examples of "Scenario Definition Languages" to fully capture complex applications with a "manifest" that could be used to produce an app and scope its behavior. This includes high level concepts that typically are found in package managers such as defining how to load "services" or dependencies, how to wire components together with message channels, how to set security policies on an application and its components, if to run components locally or in the cloud, cpu throttling and resource usage limits. A guiding principle is that even novices should be able to easily write little applications and that a third party should feel safe running code downloaded over the net. We imagine for example that a novice programmer such as say a farmer is able to wire together high level components such as a computer vision module, tensorflow, opencv and view rendering modules to produce say a farm weed zapper robot - and to then share that lightweight app easily.

2. Define common core services for an SDL. Most consumer apps are effectively a pile of services glued together. This includes networking, database support, view rendering and so on.

3. Define view and layout primitives with event handling. Often consumer apps are just list viewers - file and folder managers that often present content in a list view. That content itself is typically represented by database rows in a table on a server, and is often organized into lists on a client. We imagine this will continue to be the convention, and that there may be a focus on 3d elements as well. Here we want a simple transport notation for representing objects and typical scene-graph layouts of objects - similar to HTML. Historically VRML or X3D also had some similarities. We want to define a succinct transport grammar that lets users describe typical application concerns such as content, layout - and to let users write directly in this grammar - to intersperse macros and math and take advantage of modern language features, including callback event handlers. Effectively we see this lightweight grammar as being similar to html+javascript and or a way to effectively define lightweight applications. There should still be an emphasis on a largely declarative notation that can be "thrown over a fence" to a separate renderer that is in a different thread. This can be thought of as a layer that sits on top of a conventional 3d engine (such as 3js, babylon3d or Bevy3D) which tend to focus on exposing procedural immediate method calls to render primitives. We see a need for platform and target independent descriptions of common rendering primitives such as layout, fonts, text wrapping, animations, effects, buttons, dialog boxes, draggable regions. We also see a need for platform independent concepts around lights, cameras, 3d objects and behaviors.

Also at this time we entirely switched to javascript for prototyping from Rust and threw out the Rust code completely. We want to explore a topic space much faster than Rust is allowing us to do. We also wanted to take advantage of rich graphics rendering resources available to Electron and browser environments. Leveraging the web/javascript is hundreds of times faster for developers, so even if the implementation is not high-performance we can at least exercise the ideas and then migrate to Rust/WASM later.

Where we got to in this milestone in general:

• SDL for defining apps runs - although we find that it more convenient to express dependencies procedurally
• SDL for components runs - this works well as a design pattern and is platform independent
• Networking demo works
• 3D objects (using Babylon 3d as a target) works well and our approach should work for platforms such as Bevy3D
• Design thinking and language is maturing (the idea of a pool, services, applications and messaging channels)
• We went a bit further than just design and built tests and examples to validate the thinking

Notes:

• Philosophy - Orbital Europa Theory
• Philosophy - Why Build A Metaverse
• Philosophy - How to Build a Metaverse
• Architecture - Orbital Europa Architecture
• Practice - Orbital Europa Review
• Prototype code for this stage

Work on Orbital started at the beginning of this year with a research grant from Futurewei.

The original concept was a "blue sky re-interpretation of a web browser". Tish Shute, Yue Chen and I imagined a future where you were wearing an augmented reality display, and we asked ourselves "what kind of browser are you going to use to browse augmented reality?" The idea here is very much that "reality is the interface".

Clearly a browser for augmented reality is going to be different from todays web browsers - and yet even though day-to-day working interfaces for AR are not common, they will be in a decade. Somehow we have to go from 0 to 100% in a decade - and there has been very little work done on an augmented reality interface to the world... there has been a lot of hand-waving, but there are a lot of things that are not well defined. For example who write apps? How are those apps deployed? How do multiple apps compete for limited screen real-estate? How do you switch between apps? Do you even switch between apps at all? Or do multiple apps "decorate" your real world with augmented information at the same time? The AR paradigm is different from the 2d paradigm.

To help think this through - try walk around your house, or outside, and think about how the world around you might end up being decorated with all kinds of information -> information about when garbage is being picked up, information about super fund sites nearby, reminders to do chores, notices from the city about upcoming construction. Even things like street lights and stop signs will probably disappear from the real world and become simply augmented reality notes - all of the traditional signage of the real world that we associate with modernity are going to all simply go away because it is cheaper to render a virtual post-it note in your eyeballs than manufacture a real sign and stick it up on a street.

In the research this year we learned to see more clearly that a browser effectively is an operating system, that an index.html is essentially a manifest for producing an app, and that apps as we think of them within web pages need to improve dramatically, and that they are currently emaciated. But that also web apps have very complex security concerns and actually are more distributed than they appear (an app rarely runs entirely on device). We also learned that writing a browser at a low level is incredibly challenging.

Ongoing thinking about code architecture (version "Mercury").

_ "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." - https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule _

Our goal is not to make any given application, but to make it easy for others to make applications.

We are finding we need a higher level expression of what an app is - effectively a kind of "scenario definition language". Rust is great for writing individual high performance modules but it is entirely inappropriate for scripting glue. Rust is slow to develop in and completely inaccessible to novices (who are one of our imagined users). We find that we want to get "higher up" conceptually and expose the power to describe entire applications: how they are glued together from shared services, and where and how they are supposed to be running. Because we think we will have a distributed computational architecture we need to expose ways for users to describe the "intent" of an app. A given app may be like lego - composed out of several late binding libraries, and each of those libraries may be constrained by various security perms, and may not even all be running on the same hardware. There may be user expressed policies around persistence, storage and many other things.

We added v8 javascript to allow for basic scripting and we introduce a lightweight declarative sub-grammar in javascript we call a "flow grammar" that we use to describe most of the declarative concepts from application parts to security policies to ux layouts. By building the core capabilities in Rust, and then allowing messaging of those components from a scripting layer we are able to iterate more quickly.

This version also wraps up the initial year long development initiative.

Notes:

• Development R8 - M3 Research WebXR / WebIDL / Meta-Grammar / Graphics
• Development R9 - M3 Goals
• Development R10 - M3 Flow Language Scripting
• Development notes on event handling
• Development R11 - M3 Use Cases
• Development R12 - M3 Review
• See https://github.com/orbitalweb/mercury for the source

Ongoing thinking about code architecture (version "Charon").

Coming out of the first few months of research and thinking, we find ourselves focusing on "what is an application?".

In our model applications are loaded over the wire from a manifest. Durable state persistence is needed, and granular security. It looks like apps are defined as a collection of executable components, services or libraries with formally specified inter-module data-flow wires, with formal constraints, perms and capabilities. Individual components are separate threads. We were originally looking at dynamic binding of WASM modules to each other using some kind of IDL or ABI (as one thread) but we end up with only a messaging scheme for now (between threads).

We find similarities with existing app packagers. By defining apps as collections of components the hope is that an ecosystem of third party module developers can define components in a registry system similar to NPM or Cargo. This in turn allows device developers to ship novel capabilities faster. There's a philosophical emphasis on end users being able to wire together their own apps for their own needs.

One particular challenge is becoming highly visible - it is very hard to render the kinds of rich graphics we want in an embedded platform. It looks like modern web browsers are way ahead of embedded rendering platforms. Embeddable renderer libraries for Rust are fragmentary, very reduced feature sets, don't handle 2d layout, buttons, graphics, don't interoperate well with other libraries, have huge complex dependencies, attempt to be frameworks rather than merely services and so on. This is a big problem for us because we need good rendering and graphics to pitch our vision. We did not realize how spoiled we were with the web!

Dev Notes:

• Development R5 - M2 Governance
• Development R6 - M2 Grammar
• Development R7 - M2 Rendering
• Development R7 - M2 Kernel
• See https://github.com/orbitalweb/charon for the source code

I received the first grant from Futurewei this year. Futurewei is funding distributed computation for projects that "mature in 3 to 5 years", open source and fundamental new early stage development. In our pre-funding discussions we looked at personal computation. We see that it hard for consumers to create, distribute and run applications. We do see app stores and we do see the web as app deployment platforms today, but we also see that these tools are going to substantially change over the next decade, hopefully to become more participatory. Also, it's fairly obvious that heads up displays will not run current web browsers or apps - current apps take over a whole page display and don't cooperate with each other in the way we imagine future apps cooperating. Therefore we built a grant specifically to prototype the future of personal computation is going to be like.

It turns out that this work begins with substantial research. I thought I understood operating systems and the web well but it turns out I'm discovering that there are implicit philosophical goals that require clear consideration. For example web browsers today still have a strongly legacy focus on HTML rendering (which I consider to merely be a 'library' or 'service' capability). Browsers have a more general capability of being an app deployment platform as a whole and must focus there. As well current web browsers tend to have poor concepts of persistence and poor concepts of distributed computation. Operating systems as a whole tend to focus on "loading" and running an "app" rather than apps themselves being more like computational soups with many agents running together.

Initially we see this project as a kernel or "app runner". Thread management, messaging, packaging of apps, Rust/WASM, late binding, portability, security and sandboxing show up at this time. Use cases here are heavily focused on the idea that "reality is the platform" - that is that future computing platforms will often be 3d, computer vision, heads up display, AR and VR focused. We see use cases for perception assistance for consumers and for autonomous robotics systems (such as farm tending robots).

In this stage we did NOT focus on funding, or go to market, or commercialization. We are in a pure research mode here. We built prototypes in Rust and did a lot of thinking and reviewing of the history of operating systems and web browsers. Here are my own more detailed notes from this stage (I tend to reiterate or revisit ideas over and over here to try and frame and ground them together):

• Orbital Design Summary 2021
• Orbital Project Mercury Summary 2021
• Orbital Governance DAO Thoughts
• Philosophy/WASM Review
• Philosophy/Web Strengths and weaknesses
• Philosophy/Macrotrends out to 2030
• Philosophy/Computation
• Philosophy/Everything wants to be connected
• Development R1 - M1/M2/M3 Roadmap
• Development R2 - M1 Threading
• Development R3 - M1 Messaging
• Development R4 - M1 Manifests
• General thoughts around the evolution of the web
• Prototype code for this stage on github.com/orbitalweb/pluto