This page contains details on our Sign Language Recognition work. This work is published in the DASFAA2022 Conference and can be accessed here, and an extended arXiv version can be accessed here.

Experiments are run on five diverse data sets. Experiment data sets consist of original source files and processed files in Matlab file format. You can just download the Matlab data files to run the experiments, or if you chose to, you can generate the Matlab data files by downloading the original source files and then running Matlab code that loads in the data sets and processes them into a Matlab file format.

To obtain the American sign language KinTrans data set, you must first contact and obtain permission from linedanceAI. Once permission has been granted please contact me at [email protected] and I will provide the KinTrans Matlab data files.

The other four data set original source files can be downloaded below (files should be places in the "SourceData" folder):

• American sign language data set LM (click here for associated paper)
• Human action data set NTU-RGB+D-60 (click here for associated paper)
• Human action data set UCF (click here for associated paper)
• Human action data set MHAD (click here for associated paper)

Original source data sets are processed and loaded into Matlab. The associated Matlab files for the data sets can be downloaded below (files should be places in the "MatlabData" folder):

• American sign language data set LM files are here, here, and here.
• Human action data set NTU-RGB+D-60 files are here, here, here, and here.
• Human action data set UCF files are here and here.
• Human action data set MHAD files are here and here.

If you wish to process original source data files, then run the following Matlab code which loads the source data into a Matlab cell variable called CompMoveData:

• KinTrans - There are three xml files to import. The first step creates a Matlab data file for each XML file, and the second step loads in all three Matlab data files, determines all unique sequences, and saves the "full" set of unique sequences. Run ImportKinTransData.m three times, each time uncommenting the specific fileName to load in (Health.xml, Airport.xml, or Bank.xml). For each xml load, save results with command save('CompMoveData','KintransHealthData.mat'), save('CompMoveData','KintransAirportData.mat'), or save('CompMoveData','KintransBankData.mat'), respectively. Next, run KinTransCombineDatasets.m and then then save the results with command save('KinTransAllData.mat','CompMoveData').
• LM - run ImportLMData.m (you can save the interim results with command save('LMData.mat','CompMoveData') but this step is not required). Next, cut each sequence into segments by running LMCutSeq.m and then save the results with commands CompMoveData = CompMoveData2; and save('LMSubSeqData.mat','CompMoveData').
• NTU60 - run ImportNTURGBD2.m and then save the results with command save('NTURGBD60NoSimpSingleSubjectData.mat','CompMoveData').
• UCF - run ImportUCFData.m and then then save the results with command save('UCFData.mat','CompMoveData').
• MHAD - run ImportMHAD.m and then then save the results with command save('MHADData.mat','CompMoveData').

All *.mat data files generated above should be placed in the "MatlabData" folder.

Instructions below show how to run experiments in the paper, as well as other tools.

Information on a data set can be obtained by running CompMoveDataStats.m. Ensure that the data has been loaded prior to running this, i.e. load in the CompMoveData cell variable (You can see which data files are loaded for each of the experiments by looking at LoadModelDataset.m).

After an accuracy experiment has been run, you can view the resulting confusion matrix by running CompMoveAnalyzeResults.m.

To create a visual graph on how a particular LM sequence is 'cut' into sub-sequences, first load the uncut LM data (LMData.mat) and then run LMCutSeqGraph.m (you can change the seqID variable to see how various sequences are segmented into smaller sub-sequences).

Simple animation graphs are provided to help visualize a given sequence. The data set must first be loaded.

• KinTransAnimation.m - animation for a single sequence. You can set the seqID variable.
• KinTransAnimation3.m - animation for an accuracy experiment. Shows the query sequence, predicted sequence, and true sequence. You can set the queryResultID variable.
• KinTransAnimationFigure.m - animation used for a figure in the paper. You can set the seqID variable, and also change the seq variable to the frames you want to visualize.

• LMAnimation.m - animation for a single sequence. You can set the seqID variable.
• LMAnimationFigure.m - animation used for a figure in the paper. You can set the seqID variable, and also change the seq variable to the frames you want to visualize.

• NTURGBDAnimation.m - animation for a single sequence. You can set the seqID variable.
• NTUAnimationFigure.m - animation used for a figure in the paper. You can set the seqID variable, and also change the seq variable to the frames you want to visualize.

• UCFAnimation.m - animation for a single sequence. You can set the seqID variable.

• MHADAnimation.m - animation for a single sequence. You can set the seqID variable.
• MHADAnimationReduced.m - animation for a single sequence, with a reduced number of joints. You can set the seqID variable.

Files AnimationExample1.m to AnimationExample6.m show examples of how to animate graphs in Matlab.

The set of scripts for running the template mining experiments have the following file name format: ProcessHM<dataset>_Opt_*.m where <dataset> is one of KinTrans LM NTURGBD60 UCF MHAD. For example, to generate a feature template for the KinTrans data set, NN-s, and 2 trainers per class you run ProcessHMKinTrans_Opt_Sing_Traj_NN_2.m.

The classification accuracy experiments below already have the various feature template results contained in the scripts. For each accuracy experiment below, the accompanying template mining experiment is listed in case you want to verify the template that is generated.

The script files below run our various accuracy experiments on the KinTrans data set.

To run the ST-GCN experiments we installed their software, exported the KinTrans data from our Matlab environment into ST-GCN, and ran their training and testing code. For details click here.

The script files below run our various accuracy experiments on the LM data set.

The script files below run our various accuracy experiments on three human action data sets.

The script files below run our various query latency experiments on the KinTrans data set.

A NN-s heatmap similar to the one in the paper can be generated. First run the classification accuracy experiment. Then run NNHeatMap.m.

A DM-m heatmap similar to the one in the paper can be generated. First run the classification accuracy experiment. Then run TrainMatHeatMap.m.

The script files below run our various template speedup experiments on the KinTrans data set.

The script files below run our various feature column reduction experiments on the KinTrans data set.

The NN-s hip-only classification accuracy experiment is run with script ProcessHMKinTrans_20pct_NN_DTW_hip_sing.m, and the DM-m with script ProcessHMKinTrans_20pct_SD_DTW_hip_sing.m.