Frontend - sitcomlab/IPED-Toolkit GitHub Wiki

The frontend is the immersive part of the IPED-Toolkit which is used to create an immersion based on the data injected into the system using the backend. It displays videos and overlays them with the dynamic contents that are related to them.
The graphic below shows the frontend consisting of a video whereupon an overlay has been placed according to the semantics created within the backend.

However, since the toolkit was designed for evaluation purposes you probably want to directly interact with the contents on the screen to rearrange or transform them according to your needs. Therefore the frontend has several functionalities on board that enable you to interact with your overlays.

The graphic above shows the frontend with an overlayed 3-dimensional grid which can be used to adjust the overlays of the current location. The 3D editing mode can be invoked using the key 'i'. The following table shows the available shortcuts and their effects.

Shortcut	Effect
i	Toggles the 3D editing mode
w	Allows you to move overlays along x, y, and z axis in editing mode
e	Allows you to rotate overlays around x, y, and z axis in editing mode
r	Allows you to warp overlays along x, y, and z axis in editing mode
k	Toggle greenscreening (when using Moving Image Avatar)

While the graphic above shows the free movement mode within the 3D editing mode you can see the rotation mode in the graphic below.

The greenscreening functionality comes into play when using a Moving Image Avatar (MIA). The MIA functionality allwos it to inject webcam content into the frontend as an overlay in realtime. As with the other overlays the avatar can be transformed using the 3D editing mode. However, you can also move the avatar around using gestures recognized by a Microsoft Kinect.

(The MIA extension is currently under development. An explanation on how to make use of it will follow later as it is still work in progress.)