What are augmentations? The IEEE standard P1589-2020 defines an augmentation as the "digital representation of effector outputs that serve to stimulate the sensory experience of the user, including output to visual, audio, haptic, and other modalities". Typically, this means an augmentation is a hologram or spatial audio output, or both. Few systems currently support other modalities.

The IEEE standard defines further specific augmentation types (in the wording of the standard: so-called "augmentation primitives"). These are types of annotations, the most basic ones are: an image, a video, audio, a 3D model, or a label.

MirageXR implements all basic mandatory types of the IEEE standard, but provides a range of additional augmentation types. Most notably, this includes a GhostTrack, where a trainer hologram can be recorded from body movement and voice of the trainer to then be replayed independently by the user.

• Implemented Augmentation Types
• Implementing a new Augmentation Type
  • Register with Model, View, and Controller
    • Register the new augmentation with the Data Model
    • Register new augmentation type with the editor
    • Register the augmentation type with the Player
    • Register for saving / deleting data events
  • Programming your new augmentation type
    • Editor Panel Prefab
    • Editor View Class (Spatial UI, e.g. Hololens)
    • Editor View Class (Mobile UI, e.g. Android and iOS)
    • Controller Class

Implemented Augmentation Types

This is the full list of augmentation types supported by MirageXR:

• Audio
• Image
• Video
• GhostTrack
• Label
• Visual Effects
• Action Glyphs
• 3D Model
• Character
• Pick & Place
• Image Marker
• Plugin
• Drawing

Implementing a new Augmentation Type

When you want to extend the augmentation types available, you will have to take care of registering the new type in various places, which has to be implementated using specific classes to ensure editor, player, data storage, etc. know what to do.

Register with Model, View, and Controller

The new augmentation type needs to be added in several places: to the Data Model, the relevant classes for EditMode and PlayMode, and the relevant prefabs must be updated in the addressables mapping table[^1].

[^1]: If this is not updated for each platform (!), the build will link against the cached prefab version, causing havoc.

Register the new augmentation with the Data Model

First, you need to add a new content type to the ContentType class:

public enum ContentType
{
    UNKNOWN,
    IMAGE,
    VIDEO,
    AUDIO,
    GHOST,
    LABEL,
    ACT,
    VFX,
    MODEL,
    CHARACTER,
    PICKANDPLACE,
    IMAGEMARKER,
    PLUGIN,
    DRAWING
}

Also add a name of type string here:

    private const string UNKNOWN = "Unknown";
    private const string IMAGE = "Image";
    private const string VIDEO = "Video";
    private const string AUDIO = "Audio";
    private const string GHOST = "Ghost";
    private const string LABEL = "Label";
    private const string ACT = "Action";
    private const string VFX = "Vfx";
    private const string MODEL = "Model";
    private const string CHARACTER = "Character";
    private const string PICKANDPLACE = "Pick and place";
    private const string IMAGEMARKER = "Image marker";
    private const string PLUGIN = "Plugin";
    private const string DRAWING = "Drawing";

And add the according type description here:

    private const string UNKNOWN_HINT = "Unknown";
    private const string IMAGE_HINT = "Take a photo and add it as an augmentation to this action step.";
    private const string VIDEO_HINT = "Record a video and add it as an augmentation to this action step.";
    private const string AUDIO_HINT = "Record an audio and add it to this action step.";
    private const string GHOST_HINT = "Ghost track lets you record your movement in the real world and adds it to the this action as a virtual avatar.";
    private const string LABEL_HINT = "Label augmentation lets you add a text.";
    private const string ACT_HINT = "Pre-define models which represents verbs.";
    private const string VFX_HINT = "Visual Effects lets you add effects like fire, explosion, etc.";
    private const string MODEL_HINT = "You can import 3d models from Sketchfab.com to this action step. Sketchfab is a library with more than 3 million 3d models which half milions of them are free.";
    private const string CHARACTER_HINT = "Add an AI character to this action. You can choose between different characters that each of them can do different tasks. ";
    private const string PICKANDPLACE_HINT = "Add flags on an objects.";
    private const string IMAGEMARKER_HINT = "Image marker allows to take a photo of an object (or select a pretrained image target) and thus allow to move task stations with the marker around.";
    private const string PLUGIN_HINT = "Augmentations that are created for specific activities";
    private const string DRAWING_HINT = "Draw in 3d space";

Add the path to the mobile (screen space UI) icon:

    private const string IMAGE_IMAGE_PATH = "Materials/Textures/imageeditor";
    private const string VIDEO_IMAGE_PATH = "Materials/Textures/videoeditor";
    private const string AUDIO_IMAGE_PATH = "Materials/Textures/audioeditor";
    private const string GHOST_IMAGE_PATH = "Materials/Textures/ghosteditor";
    private const string LABEL_IMAGE_PATH = "Materials/Textures/labeleditor";
    private const string ACT_IMAGE_PATH = "Materials/Textures/glypheditor";
    private const string VFX_IMAGE_PATH = "Materials/Textures/vfxeditor";
    private const string MODEL_IMAGE_PATH = "Materials/Textures/modeleditor";
    private const string CHARACTER_IMAGE_PATH = "Materials/Textures/charactereditor";
    private const string PICKANDPLACE_IMAGE_PATH = "Materials/Textures/pickandplaceeditor";
    private const string IMAGEMARKER_IMAGE_PATH = "Materials/Textures/imagemarkereditor";
    private const string PLUGIN_IMAGE_PATH = "Materials/Textures/plugineditor";
    private const string DRAWING_IMAGE_PATH = "Materials/Textures/drawingeditor";

and the IEEE P1589-2020 predicate identifier for the data model:

    private const string PREDICATE_UNKNOWN = "unknown";
    private const string PREDICATE_LABEL = "label";
    private const string PREDICATE_IMAGE = "image";
    private const string PREDICATE_AUDIO = "audio";
    private const string PREDICATE_AUDIO_V2 = "sound"; //TODO: we should leave only one option, but we have several different references in the code. ideally, we should use the index enum
    private const string PREDICATE_VIDEO = "video";
    private const string PREDICATE_GHOST = "ghosttracks"; 
    private const string PREDICATE_GHOST_V2 = "ghost"; //TODO:^^^
    private const string PREDICATE_ACT = "act";
    private const string PREDICATE_VFX = "vfx";
    private const string PREDICATE_MODEL = "model";
    private const string PREDICATE_MODEL_V2 = "3d"; //TODO: ^^^
    private const string PREDICATE_CHARACTER = "char";
    private const string PREDICATE_PICKANDPLACE = "pickandplace";
    private const string PREDICATE_PICKANDPLACE_V2 = "pick&place"; //TODO: ^^^
    private const string PREDICATE_IMAGEMARKER = "imagemarker";
    private const string PREDICATE_PLUGIN = "plugin";
    private const string PREDICATE_DRAWING = "drawing";

Register new augmentation type with the editor

We also need to declare the new augmentation primitive also to the editor subsystem in the ActionEditor.cs class.

First, we create a new SerializeField to which we assign the according editor prefab in the inspector:

[SerializeField] private GameObject characterAugmentationPrefab;

Then add the new augmentation in spareListOfAugmentations in the We also need to declare the new augmentation primitive also to the editor subsystem in the ActionEditor.cs class.

#if UNITY_ANDROID || UNITY_IOS

    private readonly string[] spareListOfAugmentations = { "image", "video", "audio", "ghost", "label", "act", "vfx",  "model", "character", "pick&place", "image marker", "plugin" };
    private const string augmentationsListFile = "MobileAugmentationListFile";
#else
    private readonly string[] spareListOfAugmentations = { "image", "video", "audio", "ghost", "label", "act", "vfx", "model", "character", "pick&place", "image marker", "plugin", "drawing" };
    private const string augmentationsListFile = "HololensAugmentationListFile";
#endif

Please note that it is possible to overwrite this setting using the configuration files MobileAugmentationListFile.txt and HololensAugmentationListFile.txt. If this two non-tracked configuration files exist in your local branch, they define the subset of augmentations your local branch will build. To return to the default selection, simply delete these two files. These can also be conveniently ticked via the editor panel:

Then we add according code segments for instantiating the augmentation, editing the augmentation, and disabling all editors:

public void OnAnnotationAddItemSelected(ContentType type)
    {
    ...
    case ContentType.LABEL:
                labelEditor = LoadEditorPanel<LabelEditor>(TextEditorPrefab);
                labelEditor.SetAnnotationStartingPoint(DefaultAugmentationStartingPoint);
                labelEditor.Open(detailView.DisplayedAction, null);
                break;
    ...
    }

public void EditAnnotation(ToggleObject annotation)
    {
    ...
    case "label":    
                labelEditor = LoadEditorPanel<LabelEditor>(TextEditorPrefab);
                labelEditor.Open(detailView.DisplayedAction, annotation);
                break;
    ...
    }

public void DisableAllPoiEditors()
    {
    ...
    if (labelEditor)
              labelEditor.Close();
    ...
    }

Register the augmentation type with the Player

The instantiation of the augmentations happens in the object factory ObjectFactory.cs, where the info from the activity data model is processed, and the according augmentation game objects are instantiated and destroyed.

WARNING: since the IEEE P1589-2020 data model allows several ways of instantiating augmentations (directly, or attached to a 'tangible'), the way your new augmentation is expressed in the data model may vary - and, consequently, its instantiation.

You need to register the new augmentation to the Toggle method. Here is an example (REMEMBER: can vary, depending on use of IEEE P1589-2020):

private static void Toggle(ToggleObject obj, bool isActivating)
    {
        ...
        case "label":
           if (isActivating)
              ActivatePrefab("LabelPrefab", obj);
           else
              DestroyPrefab(obj);
           break;
        ...
    }

And with DestroyPrefab. Here is an example (REMEMBER: can vary, depending on use of IEEE P1589-2020):

private static void DestroyPrefab(ToggleObject obj)
   {
   ...
   case "label":
      temp = GameObject.Find(path + "label_" + obj.text.Split(' ')[0]);
   break;
   ...
   }

Register for saving / deleting data events

If your augmentation need to save some extra data as a for example JSON file you can create a method and hook it into the SaveData() method in the ActivityManager. Soon to be replaced with a new event based system.

Consequently, deleting extra data needs to be implemented for when a user deletes the augmentation. At the moment, you need to add your data deletion in DeleteAnnotation().

This will soon to be replaced with the new class system, and the ActivityController will accept registration of callbacks for the new event-based system.

Programming your new augmentation type

Editor Panel Prefab

First, create the user interface for the editor and store this as a prefab.

This user interface is for the author and producer of a new learning experience. It will list all options that the user can set.

A sensible name for this prefab is ‘augmentationNameEditor’, where augmentationName is replaced by the new augmentation name (e.g. ‘characterEditor’).

Editor View Class (Spatial UI, e.g. Hololens)

Next, we need to create the corresponding editor class. In our example here, this class is called ‘CharacterEditor’ and assign it to the CharacterEditor prefab in step 1. This class contains the common methods that all augmentations should have including close(), open(Open(Action action, ToggleObject annotation) , and Create().

close(): Close the editor panel
open(): Open the editor panel
create(): Create the augmentation

In the create() method you will define the type and the name of the augmentation. This string which will be stored on in ToggleObject.predicate and will be displayed on the annotation augmentation button (e.g. char:Alla), and also you will need it often to specific your augmentation. For example for instantiate of destroy the augmentation's gameobject. (see Step 6).

Editor View Class (Mobile UI, e.g. Android and iOS)

The new editor class must inherit from the PopupEditorBase class and contain an implementation of the OnAccept() method and the editorForType property.

OnAccept(): this method that is called when the content being created is saved.

In the body of this method you need to deactivate the old content if there was one and fill it with new data and activate it.

protected override void OnAccept()
    {
        if (_content != null)
        {
            EventManager.DeactivateObject(_content);
        }
        else
        {
            _content = ActivityManager.Instance.AddAnnotation(_step, GetOffset());
        }

        _content.predicate = $"char:{_prefabName}";
        EventManager.ActivateObject(_content);
        EventManager.NotifyActionModified(_step);

        SetupCharacter();
        Close();
    }

Old content is inserted as an argument to call the popup window:

    private void OnContentClick()
    {
        var type = ContentTypeExtenstion.ParsePredicate(_content.predicate);
        var editor = _parentView.editors.FirstOrDefault(t => t.editorForType == type);
        if (editor == null)
        {
            Debug.LogError($"there is no editor for the type {type}");
            return;
        }
        PopupsViewer.Instance.Show(editor, _parentView.currentStep, _content);
    }

You will have to set the editorForType, which is the property on the basis of which name and icon for the popup window will be set.

    public override ContentType editorForType => ContentType.CHARACTER;

The created prefab must be nested in the list of editors of the ContentListView class in the View prefab.

    [SerializeField] private PopupEditorBase[] _editors;

QUESTON TO ALEX: why don't we get this editor list now automatically from the listOfPois that now moved to BrandManager?

Controller Class

Next we create the controller class, ‘CharacterController’. This class will be assigned to the instantiated augmentation prefabs that represent the augmentation - which will be instantiated at runtime. For instance, it will be attached to the character model for character augmentation and or to the label prefab for the label augmentation. You can implement all needed methods in this class, but of course additional classes can be linked if complexity requires it.