Synthetic Humans Labelers - Unity-Technologies/com.unity.cv.synthetichumans GitHub Wiki

The Synthetic Humans package contains Camera Labelers based on Perception's extensible CameraLabeler class. These Labelers focus on human-related labeling and data exporting for post-processing.

Human Metadata Labeler

The Human Metadata Labeler outputs the properties of generated humans, including age, gender, ethnicity, height/weight, and the assets used in generating the humans. The following is an example of the metadata metrics output for a single human.

{
  "instanceId": 1,
  "age": "Adult",
  "height": "0.6",
  "weight": "0.4",
  "sex": "Female",
  "ethnicity": "Caucasian",
  "bodyMeshTag": "Mesh.01",
  "hairMeshTag": "Mesh.03",
  "faceVatTag": "VAT.02",
  "primaryBlendVatTag": "VAT.01",
  "secondaryBlendVatTag": "VAT.01",
  "bodyMaterialTag": "Female Body Material",
  "faceMaterialTag": "Head_0001",
  "eyeMaterialTag": "Eye Material",
  "hairMaterialTag": "Mesh.03",
  "templateSkeleton": null,
  "clothingTags": [
    "Mesh.01",
    "Mesh.02",
    "Mesh.03"
  ],
  "clothingMaterialTags": [
     "Multi_Colour_Weave",
     "Multi_Colour_Knit",
     "Shoe.0001"
  ]
}

Human Mesh Labeler

The Human Mesh Labeler exports the 3D mesh of each generated human into an .obj file at runtime. The .obj mesh files capture the state of the humans after they are animated and blend shapes are applied. The file paths are listed in the SOLO output JSON file.

The data exporting is expensive and is multi-threaded. Thus, it is likely for the mesh export to take a while to complete after the simulation ends. The Labeler makes sure that the simulation will wait at the end until all mesh data has been exported to disk.

The following configurations can be adjusted on the Labeler:

1. Export Mesh Triangles: This checkbox allows you to enable/disable exporting of the triangles of the human mesh to the .obj file. Because exporting meshes to disk is expensive and slow, skipping the export of triangles will speed up mesh exporting.

2. Encoding: The Labeler provides two options for different use cases.

   • ASCII: This option will write the .obj file in the standard format. Each vertex is specified via a line starting with v followed by the (x, y, z) coordinate. Each triangle is specified via a line starting with f followed by three vertex indices.

   • Byte: This option will write the .obj file with bytes instead of the regular strings. The Labeler uses Buffer.BlockCopy to convert the 32-bit float values of vertex positions into bytes. This encoding option can speed up the data export and reduce the data size, but you need to decode the bytes in post-processing. An example is shown below.

     Example byte .obj file

     Given a vertex at position (-0.05220498, 0.4371518, 0.05997515). The standard format (ASCII) will be v -0.05220498 0.4371518 0.05997515. The byte format (Byte) is \xe4\xd4U\xbd\\\xd2\xdf>\x80\xa8u=, where the first 4 bytes (\xe4\xd4U\xbd) represent -0.05220498, the second 4 bytes (\\\xd2\xdf>) translate to 0.4371518, and the last 4 bytes (\x80\xa8u=) are equal to 0.05997515.

The following is an example of the mesh metrics output for a single human in the SOLO JSON file. In this output, global_orientation is a quaternion in the order of (x, y, z, w).

{
  "instanceId": 1,
  "global_location": [
    -1.2
    0.99
    -2.6
  ],
  "global_orientation": [
     0.0,
     0.35353896,
     0.0,
     -0.9354198
  ],
  "mesh_file_path": "HumanMesh/frame_2/18.obj"
}

3D Keypoint Labeler

The 3D Keypoint Labeler captures the 3D world location and orientation of the joints in the human skeleton. The following is an example of the output for a single human. The location and orientation are both in global coordinates, and the orientation is a quaternion in the order of (x, y, z, w).

{
  "instanceId": 1,
  "keypoints": [
    {
      "label": "hip",
      "location": [
        -3.6,
         0.98,
         -2.6
      ],
      "orientation": [
        0.0,
        0.0,
        0.0,
        1.0
      ]
     },
     {
       "label": "leg_left",
       ...
     },
     ...
  ]
}