As an example of how ISETOnline can be used in a real-world project -- from beginning to end -- here is the data and computational flow we have implemented for our collaboration with Ford Motor Company on creating metric scenes for use in evaluating cameras for object detection performance and creating an extensive library of realistic synthetic scenes and measuring how a variety of sensors perform under different lighting and different image capture strategies.

Project Workflow:

• A mongoDB database was created on a network server, with keyed collections for our major object types. A Matlab class (@idb) was written to wrap basic database functionality in Matlab scripts that are part of ISETOnline. The database location and login information can be specified using Matlab prefs in a 'db' group. The function isetdb() provides a simple, persistent, connection to the current default.

• 2942 different scenes had already been created using assets created in Blender, roads created with RoadRunner, traffic created with SUMO, and object placement done using custom scripts. Each scene was transformed into a PBRT-v4 recipe, so it could be realistically rendered.

Once we created an ISETOnline datastore using mongoDB, asset information was stored using (s_storeAutoAssets.m, assetStore.m, s_assetPreview.m). Recipe information is stored using s_storeAutoSceneRecipes.m.

• Each scene was rendered several times on a high-performance compute cluster, using 31 wavelengths (400nm-700nm). Specifically, separate renders were performed for the scene's natural light (via generated skymaps based on time of day and location), street lamps for night time scenes, headlights for night time scenes, and other light sources.

• The result was a set of 32-channel HDR files in EXR format that contained the radiance from a specific type of lighting and a depth map. Since we can add light together, we were able to combine the EXR files for each scene using a set of weights -- simulating various types of lighting. When run, each scenario resulted in 2942 output ISET format scene objects. Initially this was performed manually, but can now be driven by a parameterized script using the ISETOnline datastore (t_makeScenesFromRenders.m, makeSceneFromRenders.m).

• Ground truth information was retrieved from the .EXR file data already stored in the database using gtGetFromScene.m. At this point, the metadata including scene information, pointers to assets, and ground truth object location were stored in ISETOnline for convenient retrieval and analysis.

• As an initial example analysis the ISET scenes were imaged using two different sensors and three different image capture methods (simple auto-exposure, burst, and bracketed). Each version of each image (which we call a sensorImage) was run through YOLOv4 (ol_YOLOCompute.m), with the results being stored for comparison to ground truth. By default, scenario results are not stored in the database, to help ensure only data of interest is kept.

Instead, when a researcher decides that a scenario and its results are worth preserving, they can call ******** or use ol_dataPrep.m to create sensorImages, store them in the database, and generate the data for the ISETOnline web front-end.

• As a way to allow easier access and provide a visual comparison of the results, the database was used to populate a set of preview images, exported objects, and metadata files in JSON format, using Matlab scripts from ISETOnline (ol_dataPrep.m).

• As a baseline for running future scenarios, Recipes, EXR renders, and ISET Scenes all have their data stored and cross-referenced in our database (s_storeAutoSceneRecipe, s_storeAutoSceneEXRData, and s_storeAutoSceneISETData).

• Once scenario support was added, we were easily able to run alternative lighting scenarios (daytime lighting, for example) -- using t_makeScenesFromRenders.m and ol_dataPrep.m. The results were then added to the ISETOnline web site and the scenario stored in ISETdb for future reference.

• NOTE: While not necessary, we have also adopted a more-uniform file system structure convention, with assets and other generally useful data stored at a high level, with Project specific data stored per project. Within that, Recipes, EXRScenes, and ISETScenes are all filed separately. Scenarios each get their own sub-folder for convenience and to avoid folders becoming hard to navigate.

...