The ARkit-Room Plan and ARKit Object Capture will be used to create our 3D models. The Object Capture will allow users to use their own furniture in the room models. The users will use their phone’s camera to scan their room and take images for the API calls to these model generators. The front end of the application will allow users to modify the room layout plans based on their preferences which will be viewed using the visualization framework. We will use our secret sauce algorithm which the front end will call to provide the recommended layout to the user. The user will then be able to modify this layout plan and save it for future usage in the backend of our application. We would be able to upload these models to the cloud and later users could also retrieve these models from the cloud.

which integrates each component and collects user data

1. The frontend displays dynamic preference questions to the user, and saves user preference data for the generation algorithm
2. Once the user requests to generate layout recommendations, the frontend calls the generation algorithm to provide recommended layout to the user, and forwards the corresponding user preference data to the generation algorithm, according to its API design
3. After the user scans or uploads the room layout, the frontend forwards the 3D model data to the visualization framework to allow the user view the layout, according to the API design of the visualization framework
4. Allow user to edit and move furniture based on SceneKit

which renders a interactive 3D model based on usdz format model/data input.

We need a visualization framework to

1. render the original and the generated 3D model of the room
2. allow users to move, rotate or delete the furniture objects

We plan to introduce a 3rd-party SDK, SceneKit to realize the visualization framework.

The main classes we will use in SceneKit is:

1. SCNScene: A container for the node hierarchy and global properties that together form a displayable 3D scene.
2. SCNView: A view for displaying 3D SceneKit content.

There are two kinds of inputs we need to deal with:

1. 3D object (.usdz)

For the input of 3D object, we will seperate the whole model into two parts: wall and floor. We will consider them as two nodes and append it to the current SCNScene. For windows and doors, they are appended to the subnodes of wall. For other furniture objects, they are appended to the subnotes of the floor.

2. config files describing the location information of room and furniture (.json)

For the input of config file, we create a new SCNScene, and then read the tag of each entry in the file and append them to the SCNScene according to the rules mentioned above.

Once the User Frontend sends a json containing furniture dimensions, the pre-generated 3D room model, and the user selected preferences to the layout algorithm, it executes a decision tree of actions in response to the given criteria and to determine the optimal layout. Then, the algorithm associates the empty room with a grid system, and assigns all furniture corresponding locations in the grid system based on the generated layout. When done, the data is sorted into a results json containing all the necessary parameters of the recommended room plan back to the User Frontend before entering the modeling stage.

which generates a 3D model of users’ room via scanning

We need to use the Room Plan to help us

1. Generate direct 3D models of users' rooms so that we can display a previewed room to users using RoomCaptureView The class provides

• A live camera view that allows users to observe their room with augmented reality features.
• A real-time camera display that lets users see their room enhanced with augmented reality elements.
• Guidelines that clarify how to place the device, especially if the system needs a particular device motion or viewpoint for successful capture.

2. Obtain detailed output as USD or USDz files which are adjustable using other 3D modeling tools

Object capture will allow users to create 3D models of the furniture they already own. This furniture will be used in their AR models for the most realistic experience.

• It takes a set of photos from multiple angles of an object and turns them into a 3D model.
• You submit the collection of images to RealityKit using a Photogrammetry Session
• Object capture then generates a USDZ file as the output.
• The session uses an async sequence to deliver status updates that you should register to.
• PhotogrammetrySession.Configuration can be modified in order to have custom configurations such as image contrast for a better 3D model.

This function will be used which allows remote storage of plans and user data

• Set up API endpoints for communication with the Frontend (such as Generation Algorithm, ARKit Room Plan, and ARKit Object Capture components)
• connect Google Cloud Endpoints for cloud storage services.