Extracting Faces and Fixing Alignment - MachineEditor/MachineVideoEditor GitHub Wiki

One of the main goals of MVE is to ease data preparation, and that starts with getting Face images, curating the set, and fixing problematic alignments.

---

this step is basically the same if used via DFL bat files, or via MVE, except MVE provides a nice graphical interface for option selecting

After setting up your work directory (File > Open Folder), the first thing you need to do is open your Frames (usually extracted from your Video file) by clicking Open Images, like in the picture below

Once your Frames are open, click on the 2nd button from the top in the side-menu on the right (Detections):

1. Detections button
2. click to select the directory in which your Faces will be extracted (that is usually \Aligned)
3. select face type/size (Half Face, Full Face, Whole Face, or Head)
4. type in the desired resolution for extracted faces, or leave default value of 512
5. type in the desired jpeg quality, or leave the default value of 100 (max quality)
6. type in the maximum amount of faces to be extracted per frame, or leave 0 for "all possible faces"
7. select whether you want to run extraction on your CPU or your GPU
8. once all settings have been selected, click Extract

This action will usually take a lot of time, especially if you have a lot of Frames.
This action will also quite often result in many unwanted or low quality Faces, which you may want to curate by using something like Face Similarity sort

---

After you have removed unwanted pictures from your Faces, it is time to set them to their Parent Frames.

1. open Detection Management from the side-menu
2. make sure Face is selected from the dropdown menu
3. if Parent Frame folder is not already set, click to point to the correct directory
4. put a checkmark on Set faces to parent frames
5. once all settings have been selected, click Import face data

This will take some time, but it is much faster than Extracting Faces.

DFL's face extraction some times results in the same face being extracted more than once from the same frame. It is also possible that another face (ie not from your subject) slipped through and is now set on the Frame.
In such cases, there's a relatively easy way to sort and remove those:

1. Open your Frames (after you've set Faces to Parent Frames)
2. click on Filter and select Multiple Faces
   
3. right click on the Face you want to keep and click Delete Others

Repeat 3. for each frame with unwanted extra faces before proceeding to the next step.
Once you're done, you should click the Save button at the top.

---

Important notice: it is currently (up to and including MVE 0.9.0 which is currently the latest version) not possible fixing 3D alignments used by Head. This only works for 2D alignments used by WF, FF, etc

There's two main ways to fix bad alignments, with slightly different uses:

• Using Alignment Editor - primarily for smaller tweaks
• deleting and then copying a face from one of the neighbors - primarily for really bad alignments

Important: After you are done fixing alignments, make sure to save your changes by clicking the Save Button (floppy icon) at the top

To open Alignment Editor

1. open your Frames (after you have set Faces to them)
2. right click on the face inside the green bounding box
3. select Edit Alignment from the menu
   

Alternatively, in versions 0.8.0 and later you can CTRL+click on a face inside the green bounding box, and that will make Quick Editor go from preview mode to editable mode.

to disable Quick Editor Preview (versions 0.8.1 or later) from showing up when you hover over faces, go to Alignment Editor and remove the check mark from Show Quick Editor on Hover

You can now manipulate the Alignment dots in three ways:

1. individually (drag & drop functionality)
2. as a group
3. or by using a network model

First two of those options are directly available inside Quick Editor.

To select multiple alignment dots as a group, you have 3 basic options:

• hold left click and move the mouse to select all dots within the square you draw
• holding SHIFT + (CTRL) + clicking on dots
• use the buttons or keyboard shortcuts for predefined groups (All, Outer, Nose, Mouth, etc)

Picture showing predefined groups, and the option for selecting different colors for different groups (available from 0.8.2):

hint: you can see keyboard shortcuts when you hover over one of the options, or by clicking Shortcut Help button in the bottom of the screen

When alignment of a face is really bad, it might be worth completely removing it, and then copying it from one of the neighbors. This can be especially useful for obstructed faces (if neighbors aren't obstructed).

The deletion and copying of the faces can be done both from the main screen, or from within Alignment Editor.

If you want to remove a face from the Alignment Editor, click Delete button from the bottom, or use keyboard shortcut Del.
To remove a face from the the main screen - right click a face (inside the green bounding box) and select Delete.

please note that there is also Delete Image, and that one would permanently delete your Frame

To copy a face, there are several options:

• [main screen] right click on a frame with face, select Copy Face, right click on the frame where you deleted face > Paste Face
• [main screen] right click on a frame where you deleted face > Add Face > either Previous frame or Next frame to copy from that one
• [main screen] right click on a frame where you deleted face > Approximate faces from neighbors - this one requires both previous and next neighbor to have a face
• [alignment editor] open a neighbor frame's face > click Copy to Next or Copy To Previous as needed

After you copy a face, you should check it out, it may need some fine-tuning.

short video showing the full workflow for alignment fixing - includes most of the steps, from setting to Parent Frame, fixing alignments, and then Exporting fixed Faces

alignment helper and Quick Editor - shows some of the new functionality from MVE 0.8.0 and later

---

Once you are happy with the alignments, it is time to export Faces. That way, newly exported Faces will have proper alignments and be better suited for Training.

1. open Detection Management from the side-menu
2. make sure Video frame is selected from the dropdown menu
3. click to select the directory in which your Faces will be exported (that is usually \Aligned). Warning selecting a folder with existing images will delete all images before Export starts
4. type in the desired resolution for your Face images
5. select face type/size (Half Face, Mid Face, Full Face, Whole Face, or Head)

it is also possible to use Custom Exports

6. type in the desired jpeg quality, or leave the default value of 1 (max quality). 1 here is the same as 100 in step 5 of Extracting Faces
7. once all settings have been selected, click Export faces to images

This will take a lot of time, but it should be faster than Extracting Faces.

Starting with MVE 0.8.2, it is also possible to fix/export a face directly on the Face image (as long as it has Parent Frame info, and Parent frame is available) - you can see that showcased in this video

---

• Commonly Extracting is the initial Face image creation and Exporting is from Frames that have had Faces set on them via Set To Parent Frames, but many use the terms interchangeably
• Resolution in all the settings here asks for a single number between 256 and 2048. The actual resolution of the picture will be the same in both dimensions (ie 512 x 512)
• Heads use 3D alignment, all other Face Types use 2D alignment - because of that, if the original Extract was for Heads, you won't be able to Export anything other than Heads from Parent Frames, and vice versa: can't export heads if parents had a face with 2d alignments set