Creating Great Looking AR Models - juniverse1103/ARKitStudy GitHub Wiki

Creating Great Looking AR Models

Omar Khan, Product Design

Essential Concepts

Essential concepts to keep in mind while developing AR experience.

People's expectation

• Fast and consistent rendering performance
  • Maintain 60 fps rendering throughout the entire experience.
  • Make sure that the performance does not ever degrade in every possible angle. Test many times til it gets perfect.
• Minimal battery impact
  • Make sure that the AR experience is able to have as minimal of an impact to battery life as possible.
• High fidelity assets and cohesive style
  • People want to see all the nuanced details and explore features.
• Seamless blend of AR and real world
  • People want to use AR in a wide variety of environments.
  • Avoid the contents stand out in real world locations where potentially the lighting ambient conditions could conflict with the intention.
  • Try to avoid using colors that are either too bright or too dark.
  • Light the AR scene in a way that it casts even lighting on all the objects that you're planning to render, and at no matter what angle you view from.
  • Make sure to able AR content to work whether it's day or night.
• AR Quicklook
  • View AR assets on iCloud Drive using Files app on iOS, and project it into AR with AR Quicklook, one of the newest features on iOS 12.
  • Safari is also available for AR Quicklook.

Planning the AR experience

• Number of objects on Screen?
  • Does it need to render hundreds of AR objects, or will we focus on a single hero asset?
• Graphical style and details?
  • How much detail do we actually want?
  • What graphical style best represents what we're trying to convey?
• Level of interaction?
  • Level of interaction we want people to have with our experience?
• IKEA Place app
  • Different furniture pieces are the main assets.
  • Present highly detailed objects that closely mirror the real world counterparts.
  • In this case, rendering quality of assets are the most important.
• Bricks app
  • Potentially having a lot of objects being rendered and interacted with.
  • Many simple low-poly models witha. aflat colorful material.

Setting up your AR canvas

• Create a focus square to determine where to start place AR content.
• Statistics panel allows to see current frames per second, as well as how many polygons there are visible on the screen at any given time.

CampfiAR

• Intro

  • Sytlized, semi-realistic, and playful graphical style.
  • Unique details with physically based materials

• Working with meshes

  • Mesh, the foundational structure of 3D objects.
    • First thing to do is to lay out the basic structure of an AR scene by using simple meshes.
    • Mesh is a collection of triangles that are arranged in 3D space that will form a surface for you to apply materials on.
  • Typical development flow that will allow to create highly detailed models
    • Try to reduce the number of polygons with 3D authoring tool's function.
    • Maintain 60 fps.
  • Producing great looking materials
    • Physically Based Rendering(PBR)
      • Apply different material properties onto the mesh.
      • React realistically to the simulated lights found in AR scene.
      • Refer Advances in SceneKit Rendering, WWDC 2016 talk.
  • Albedo
    • Referred to as the base or diffuse color property.
    • Albedo is the material property that typically used to apply textures to the surface of model.
    • Mesh contains different verties, that held different pieces of information.
    • UV coordinates help map out how pixels from various texture maps are actually applied to the model.
    • You never know where people are never going to experience your app.
    • Have your content fit in as many different scenarios as possible.
    • Don't select too bright or dark albedo value.
  • Normals
    • Normal maps allow to add fine surface details back into models, without the need to add any additional geometry.
    • No changes in the number of polygons.
    • Generate normal maps by projecting details from a high-density mesh over to a low-density one with 3D modeling apps.
    • Colors of the normal map is visual representations of the vector data.
    • Determining how the normals on the surface model will be offset in order to change the way that light is being reflected.
    • Normals are the normal vectors that lies perpendicular to the mesh
    • Why normal vectors are important?
      • In order to see our object, we need to place simulated lights into our 3D engine.
      • Normal vectors allow 3D engines to calculate how these lights are actually reflected off the surface of these materials.
      • Smoothe or phong shading calculates the reflection off of this model, giving the impression of a smooth, curved surface due to the gradual interpolation of normals.
      • Two models have the exact same number of polygons!
  • Metal map
    • Shiny or Reflective objects.
    • It doesn't necessarily mean that the object needs to contain metal. Metalness is just letting the 3D engine know that the object should exhibit a reflective surface.
    • Use metalness map when it has both a mixture of metallic and non-metallic surfaces.
    • Metal map is a map represents metalic surfaces being black, from non-metal as white.
  • Roughness
    • Use roughness to simulate micro surface details, which in turn will affect the way light is being bounced off that surface.
    • If the roughness property is set to completely smooth, the light will bounce off the surface like a mirror.
    • As increasing the roughness of the material, light reflects over a wider range of angles.
    • Roughness map is a good way of being able to simulate the concept of having microsurfaces, and blurring reflection to the point where you might not see any reflectivity depending on which range you put your value to.
    • Combination of metalness property and roughness property make reflective AR models look phenomenal.
    • Roughness can be used to tweak how much of the objects will reflect the environment.
  • Ambient occlusion
    • Providing model with self-shadowing, which in turn can lead to adding additional depth and details to AR models.
    • Apple recommends to bake developer's own ambient occlusion into a map rather than use the alternative.
  • Transparency
    • When using transparencies, Apple recommends to separate materials for objects where we see a combination of transparent and non-transparent surfaces.
    • When working with AR content, the use of a lot of transparent surfaces can potentially have a huge impact on performance.
    • Especially in a case known as "overdraw", which transparent surfaces are stacked in front of each other in a view.

• Grounding AR content to the real world

  • Light estimation
    • Compensates varied lighting conditions.
    • The way this works is that ARKit analyzes each frame of the video, and uses them to estimate the lighting condition of the real world.
    • This magical feature helps assure that the amount to light applied to the AR content matches what we see in the real world.
  • Shadows are hard.
    • Shadows are a great way to make objects actually feel grounded in the world, but misplacing shadows might make it worse.
    • Apple suggests to place directional light directly overhead, and play a little bit around with the intensity of that effect to make sure that it actually feels a little bit more subtle.
    • Alternative is to create own drop shadow, rather than using dynamic lights in the scene.
    • Take the time to craft the shadows.
  • Environment maps.
    • Use Environment maps for AR objects that exhibit reflectivity.
    • Environmental maps are used to simulate the ability of metallic surfaces to mirror the environment around them.

• Things to Keep in Mind

  • Real world environment considerations

  • Maintaining your rendering budget

  • Leverage the benefits of physically based materials