Form versus Function - General Thoughts

We as an industry seem to focus on appearances rather than substance. I think we do the technically easy thing (even though it can be hugely more menial labor to make something pretty rather than functional) - and avoid the slightly harder thing (to make things actually responsive).

For example we see a fixation on static visuals in Google Maps. We do not show vehicles or people or any interactions. Cell phone data does exist for many users, and many users would share it - so why is google maps such a desert? Why is there so little focus on the most interesting thing about digital twins and about real places - which is the people in them?

As well historically we've had web servers that used to ship static content, although now they ship apps, but those web servers still have a huge focus on layout - and I would say that layout is burdening down or inhibiting what they could actually become - which is pure app containers. The original legacy of the web was a focus on layout and aesthetics - effectively visuals - and this has become an anchor.

We also see the same with the metaverse - many metaverse ventures are focused on high fidelity visual graphics. I do think as a maxim at this point that focusing on visuals is company destroying. If you are Electronic Arts and you have an AAA budget you can deliver superlative graphics and in a sense out-compete other game developers, but if you are not EA you probably should not try to compete in that one area.

At the same time, ventures that focus on AAA graphics do struggle to also deliver rich behaviors, especially participatory rich behavior. Unreal for example delivers rich graphics, but it takes a separate company (Improbable) to try turn it into an MMO engine with persistence and scaling - and that separate company burned through 500 million dollars trying to deliver this vision.

Existing engines start out with a graphics focus, but often find their value equally or if not more in being a framework. Bevy3D being a good example. The ECS model in Bevy are hugely valuable.

We do see many games such as https://veloren.net/ and https://palia.com using stylized views which does reduce their computational burden - in their attempt to deliver on MMO style features.

Orbital in my mind effectively provides value by becoming a place where people can share behaviors - a kind of behavior zoo. In fact it may be both playful and serious - in that some people may build light entertainment experiences out agents in the Orbital ecosystem, while other researchers and developers write new agents. I could see cut, copy, paste and edit of agents and agent pools. This is not dissimilar from Docker, which arguably isn't the only OS container, but because they have a large content repository they continue to remain a valuable resource.

Currently I am focusing on a single demo - an urban city model with some kind of vehicle model. This is a simple well understood use-case and can both be fun and serious. But to accomplish even simple vehicle behavior is not trivial. Even if we take perception for granted the decision making and behavior can be complex. Finding good solutions for vehicle behavior may require a large pool of contributors. Hugging Face is a good example of a socially driven community that is both playful and research oriented. And Midjourney also has a similar kind of social activity on discord.

It's not entirely clear why Unreal and Unity or some of the larger game content development systems don't have more vibrant app stores in this way. They are successful, but I would expect thousands and thousands of agents simulating everything from ants to elephants. Something more like SketchFab but for behaviors. And not necessarily only visual behaviors of visual systems but of all kinds of systems. Is there a lack of a shared physics foundation? Effectively I imagine a game engine would emerge that is AI centric. We do for example see in the latest NVidia 4090 a heavy emphasis on AI - so why is this not percolating out to agent zoos yet?

With a good game, or set of games, there would be lots of players, this would help provide good ground truth and good datasets, and then researchers could test models on top of that, and then those models would improve user experience, and this would be a virtuous feedback loop.

See

• https://www.improbable.io/
• https://www.wrld3d.com/
• https://hadean.com/
• https://meetkai.com/
• https://maikklein.github.io/unreal-rust-1/
• https://medium.com/embarkstudios/embarks-creative-playground-call-for-early-players-2e1d56d21ede
• https://medium.com/embarkstudios/our-continued-journey-89dad7a76bff

For me this means I should focus both on agent zoos and on letting people play with their own sims, and to cut, copy, paste other sims.

Milestone 4 Goals

To prototype a car simulator as a test I break it down into a few pieces:

1. First getting the simulation itself right - pedestrians, vehicles, independent agents per pedestrian and so on. This does not need graphics, does not need networking, does not need real world data. Our priorities are very different in that we focus on real behavior - not on pretty graphics first.

2. Then we focus on pretty graphics. There are third party models of buildings that we can use. We will not explore procedural generation of geometry (because it is not critical path). We want pretty graphics because it helps communicate the ideas.

3. Then we focus on correctness. We want to use the SDL notation to describe agents, and how they are loaded into the compute sandbox. Client side views should be able to be generated on this. We should be able to do live programming, and iterate extremely quickly on evolving the experience if we wish. This starts to deliver on the idea of a "browser" effectively being a window onto shared computational resources and an idea of participatory authoring, shared reality and where we think the future overall is going.

We do not especially dig into computer vision for this iteration. Unlike CARLA itself we are not focused on CV here - we are instead focused on heterogenous agents and a distributed computation model with sandboxes that are safe for agents written by different parties to run in.

Milestone 4 General Thoughts

Because of my current focus now on a small city - I am looking at various ways to simulate traffic - and to procedurally generate entire cities themselves as well. My focus through to the end of this entire year will be to make the best possible urban city model that is possible. It will be interactive and multiplayer and should hopefully be at least moderately interesting to play. Perhaps the only other thing I will try stuff into this is shared voice so that our discussions about the project can be done in the project.

I have two active simulations so far - for pedestrians I have a simple weekly schedule they follow, and for vehicles right now they just go from point to point randomly. I've been using an a star algorithm for navigation but I probably will switch to a navmesh.

The final pedestrian model I am thinking of is something like this:

• pursue daily goals; such as going to work during the week day, and parks or other activities on the weekend
• generally if far from a target, and does own or can hop in a cab then take that vehicle to the target
• generally if near the target then either try park the car or hop out and continue on foot
• and otherwise if near a target then walk to it

Vehicles I think will have something like this (they already do some of this now):

• Navigate from point to point - probably using an a star algorithm although it isn't very flexible; navmesh may be an option
• Model engine acceleration
• Model street lights
• Import road networks from the map itself (right now I build them tediously by hand)
• Refine collision model a bit
• Deal with parking spots and have some kind of unoccupied parking spot strategy

I feel like this could be good enough to actually be useful. Another option is to find third party agent models for vehicles and use those.

Here are a few links:

https://github.com/eleurent/highway-env https://www.youtube.com/watch?v=7aP_QAypcBM https://github.com/mikewesthad/navmesh -> this is a standalone tool that can do polygon based navigaton - stable? https://github.com/qiao/PathFinding.js -> an old pathfinding lib -> i do need agent collision support https://github.com/donmccurdy/three-pathfinding https://github.com/recastnavigation/recastnavigation https://redblobgames.github.io/circular-obstacle-pathfinding/ https://npm.io/package/babylon-navigation-mesh https://hubs.mozilla.com/labs/creating-a-custom-nav-mesh-for-your-hubs-scene/ https://mugen87.github.io/yuka/