Scripting -> flow

We have an emerging scenario description language we call "flow" for describing entire applications, their components and how those components are wired together, their security policies. This acts as a portable manifest or production grammar that treats the entire Orbital Engine as simply a service that produces whatever the grammar specifies. This also encourages late binding, and also encourages dynamic distribution of app components over networks as needed, rather than historical app patterns.

This is quite different from the "monolithic" app pattern that we see on iOS, Android and other operating systems. The current application paradigm is to describe your application glue in a native language and then compile that glue and the components down to a single binary blob. This legacy pattern exists because of a historical emphasis on performance on underpowered machines that were not able to resolve or finalize inter-component bindings by themselves. Today this is less critical, and we already see WASM/WASI and late binding patterns that reduce the value of a "compile everything" mindset. As well, in a modern computational soup an "app" itself is distributed over multiple machines, and sometimes load-balanced on the fly; so increasingly there is a tension with monolithic app patterns.

Here are some examples of high level scenario or system description grammars:

1. https://carla.org describes rich vehicle ecosystems
2. https://doc.rust-lang.org/cargo/ describes components of a rust app
3. https://www.safetypool.ai/
4. HTML itself, as a manifest that describes a rich app on the web

If we look closely at Orbital we have to think about these goals:

1. How do we measure success of this project? A successful design metric is supporting these actions: Supporting Embodied Programming, Live Programming, Creative Programming, Social Programming. -> A design goal I want to support is to make sure that end users can easily wire together their own apps, that this can be done live, and that it can be done in real time with friends. Later I want to support visual wiring of objects together to support embodied programming. A metric for success here is that users should be able to work within this framework and not get physically exhausted by it. Many frameworks work, but many are also overly complex and stressful.

2. Firm up / Promote the Metagrammar. We already have a burgeoning grammar that tells the system which components to bring up, what the security is between components, and how to wire components together. That grammar is turning out to be effectively a graph or tree with a single namespace. Users mount or unmount fragments of the graph and the system state is a reflection of that graph. All existing services are in the graph. All user accounts are in the graph. All views and desktop layout is in the graph. Security is described in the same graph. It looks like a kernel or operating system isn't hard to define, but driving one does require setting conventions, and this grammar is used to describe the pieces that come up and exactly how they do so.

3. An idea of a hypervisor is emerging. In my system an "app" is a few components wired together that are messaging each other. They exist in a general computational soup. A hypervisor can load balance parts of apps to different computers around the net based on different criteria such as user latency, speed, price and security.

4. An idea of semantic intent is emerging. Often a component will want to open a socket or open a file to do a raw low level operation. If components can instead express a higher level semantic intent then a brokerage becomes possible where arbitrary other components can resolve the need. For example if a component says "I need to known the weather in PDX in the last few hours" that request can be resolved by it opening a socket to weather.com, fetching some json and parsing it - or perhaps some other component can also resolve that request, or has recently resolved the request. Effectively components could exist in a kind of "chat room" and could have a higher level conversation about need resolution.

Where scripting is showing up forensically in this app:

1. Javascript is being used more heavily now. Considered https://github.com/rhaiscript/rhai and https://www.assemblyscript.org/ but ended up sticking with V8 - which is a pain but works. The metagrammar runs inside of the scripting layer for now. There are many versions of v8 bindings. I am using rusty v8.

2. Components need wiring. I did revise the kernel architecture a bit, it is largely the same but there is a more formal concept of a single namespace that components (or services as I sometimes call them) are registered. It is worth noting that components still message each other using channels on a pub/sub bus architecture. This is similar to https://dapr.io/ . Also worth noting https://wasmedge.org/ . I'm still sticking with my own solution however since I am doing a blue sky product. Later on I can revisit.

3. Display needs a way to express widgets.

Flow Language

There is a burgeoning grammar that is being used to drive the system. I call it "flow".

This how the system is running now using this language:

1. First, before any grammar is run the system had a hardcoded startup. The first piece that is loaded is a "broker". Every other component registers with the broker. Any component can ask to listen to a specific message channel, or can publish to a specific message channel.

2. Permissions are generally handled by the broker. When a component is registered all the perms are registered with it. All messages flowing in or out of components are mediated by the broker security policy on that component. There is no way to communicate with the world other that through messages. The filesystem and all other services are also components.

3. I have a few hardcoded (rust native) components that can be mounted at will. This includes a display driver, a scripting driver, a wasm loader, a tensorflow capability. There is also a built-in broker that is capable of receiving messages (using crossbeam) and routing them to registered components.

4. The built-in components all happen to have "factories" - or basically a capability to instance others of themselves. The scripting component can spin up a script, the WASM component can produce a separate WASM component from a remotely fetched blob. I found it was easier to statically register components and to then delegate the work of having factories to those components rather than introducing a separate factory concept. This keeps the system simpler.

5. In rust I fire off a scripting instance of a boot.js file.

6. In the boot.js file I first introduce conceptual abstractions of lower level rust-side machinery. It contains ordinary javascript methods that invent an idea of a "service" that you can query to talk to the rust-side kernel. So we have (on the javascript side) what appears to be an interface to the rust-side. These methods actually parse "flow".

7. All "apps" are a single file that is a "manifest" and that cite other components and describe how to produce them, wire them together and their security. This is identical to the role index.html plays in a browser. We also have a DOM like quality in our manifest similar to HTML. A manifest is a declarative graph in flow. Flow is basically just a hierarchy of nodes with properties. You describe (declaratively) fragments of a graph, which are then piped to the rust-side and are used to form a total system state for the entire machine. The rust side makes sure that the actual live system corresponds to the graph.

8. In javascript land there is a bootstrap "application" (a manifest defining a fragment of a graph) described in flow. It describes what child components to bring up and what to do. In the case of booting up the boot manifest says that it wants the display component to exist, and that it wants to paint a login display (with buttons, text, images and so on) into that display engine. It also has callbacks for triggering state changes to the desktop display.

9. In general user events are piped up from rust-land into javascript and then caught by userland event handlers (such as button handlers). In the case of the bootstrapping, when you click on the visual "login" button this invokes a callback up to javascript which then orders the display component (down in rust) to throw away the login view, and replace it with a desktop fragment.

10. The desktop fragment is in charge of giving the user a sense of control over the system. It lets the user see what is running, start and stop things and so on. It is able to ask the rust side for a current collection of running apps and then it paints them visually for the user to feel like they can an enumeration of what apps are running. Apps themselves are also talking to the same display and are also painting their own graphics to the same raster. In general the desktop watches for user activity and can handle user requests to start or stop existing apps, and to register and run new apps fetched over the wire.

11. It is worth noting that both the description of an app (as a collection of components wired together with security policies) AND the description of a view layout are BOTH in the same notation.

12. The display grammar does have a lot of subtle expression to perform sophisticated rendering of text, boxes, images, 3d objects, cameras and lights. These are not all implemented yet however, but they are 'defined' - and remain to be filled out completely. The display notation also introduces an idea of prototypical visual elements that can then be subclassed in order to allow style-sheets. The prototypical visual elements are themselves also in flow - and you can see them here.

Scripting grammar Notes

The goal is to formalize the largely declarative grammar that makes it possible for novices to drive orbital; to set up modules, wire them together, handle events. This may end up being the core of the entire system however - since a grammar like this ties everything together.

Example Script

For example this is a typical document that produces a camera face segmentation app

let myapp = {
	camera:{
		kind:"modules/display/camera",
	},
	segmenter:{
		kind:"modules/face_segmenter",
	},
	display:{
		kind:"modules/display"
	},
	wire1:"camera -> segmenter",
	wire2:"camera -> display",
	wire3:"segmenter -> display",
	sponsor:"@dorothy",
	copyright:"MIT license",
	on_event:function(ev) { console.log("lifecycle event called"); console.log(ev) }
}

A few design choices

General lightweight vanilla json -> basically just use a text declarative-foremost json-like grammar where perhaps the "edges" can be procedural, but the bulk is just declarations.

Crumpling -> I did decide to still support some convenience concepts such as "crumpling"; which is an idea that the grammar in text doesn't have to fully qualify some relationships such as parent { children[ child {} ] } but can just do parent { child {} } if the user wishes.

Manufacturing by message -> As nodes are parsed I effectively turn them into messages, directed at any appropriate handler a section of a graph; so something like root { mydisplay { kind:"display", mylight { kind:"light" } } } would be instancing a node of kind display, and then passing the light to the display (to make if it wants). Each leaf handles the manufacturing of things below itself at will and returns some kind of UUID (or one is shovelled down with the node). Note that it is so much easier to work in JS that I actually write this piece of parser logic in JS itself.

NO ECS -> There is no formal ECS model - objects can have children objects that implement a behavior but there is no separate "component" list per object.

Events -> There are lifecycle events fired off to "on_event" that can be helpful (procedural logic is allowed on object declarations).

Special props on nodes

1. "id or uuid" -> This field is reserved internally

2. "name" -> All objects must be named, but the name can be defined in the named hash entry if desired (rather than in the name property).

3. "copy" -> This specifically mixes in any fields of the previous objects specified

4. "kind" -> this specifically binds them to internal system capabilities code or logic if defined

5. "children" -> this specifically is recognized and used by a filesystem like query capability

6. "on_event" -> procedural logic can be attached as to objects that is triggered on certain transitions

7. "on_change" -> what do we want to do with prop events should we pipe them to the object?

Ongoing grammar research

Simplify/Revise Manufacturing ?

In github.com/anselm/blox we had a pattern of named entities such as this: { mesh: {} }
This allows only one of a thing in a parent
Instead do something like { named001_mesh {} or named001 { kind: "mesh" }

Dealing with callbacks?

- there are some hassles around callbacks and messaging and so on with rust
	so i think the quickjs call back basically needs to use messages to comunicate with any state or data
	so those messages talk to something else that can deal with tne node creation events; routing as needed

- i'd love to write more code in quickjs, notably promises and so on, and really drive the system
	it is not perfectly clear how to do that

Allowing includes ???

Orbital scripting notes

I have the option to register callbacks to rust or I can build entire interfaces in javascript. I may as well just stay in js for early tests.

I can also emulate a web component registry https://blog.logrocket.com/what-happened-to-web-components/

It's worth noting that the html5 spec is grossly stretched by WHATWG https://en.wikipedia.org/wiki/HTML5

Constraint Grammar

The meta-grammar or mini-grammar also should not just capture relationships but also define capabilities/perms. I see several roles in fact:

1. State transport and Interface Definitions. For example myfunction(int a,int b) can throw an error, and is asynchronous, and returns a promise or a float. It might even be nice to specify time limits.

2. Permissions; to specify if one module can call a method of another module basically.

3. Limits; to specify how many resources one module can ask of another.

4. Capabilities; usually based on a key or permission [ https://en.wikipedia.org/wiki/Capability-based_security ]

5. Semantic Intent; for example "place this photo on a wall" as opposed to "place this photo at xyz"

There are different ways different people talk about this idea:

1. WASM Interface Types. https://radu-matei.com/blog/wasm-api-witx/ . This way of thinking about how modules talk to each other is based around an idea of "capabilities". Given modules can export an API and others can consume that API and an intermediary may set constraints or limits. See also https://www.tutorialspoint.com/cplusplus/cpp_interfaces.htm . Also see https://hacks.mozilla.org/2019/08/webassembly-interface-types/

2. WASI. This is a sloppy idea for formalizing how WASM modules talk to a "system" which is a presupposed magical set of capabilities that exists outside of WASM modules. https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/ . The conceptual defect here is that the "system" really should just conceptually be another module, not a magical thing with special hard coded properties.

Other discussions around state transport between WASM modules

https://alexene.dev/2020/08/17/webassembly-without-the-browser-part-1.html

https://www.youtube.com/watch?v=vqBtoPJoQOE

https://docs.wasmtime.dev/examples-rust-hello-world.html

https://docs.wasmtime.dev/examples-rust-wasi.html

https://docs.wasmtime.dev/examples-rust-multi-value.html

https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

https://labs.imaginea.com/talk-the-nuts-and-bolts-of-webassembly/

https://kevinhoffman.medium.com/introducing-wapc-dc9d8b0c2223

https://github.com/wasmCloud/wascap

https://github.com/wasmCloud

https://www.ralphminderhoud.com/blog/rust-ffi-wrong-way/

https://doc.rust-lang.org/nomicon/ffi.html

https://www.youtube.com/watch?v=B8a01m8B6LU

https://rise.cs.berkeley.edu/projects/erdos/

https://www.w3.org/2018/12/games-workshop/slides/08-web-idl-bindings.pdf

https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/features/ux-building-blocks/app-bar