Dataset and Structure - thuiar/MMSA-FET GitHub Wiki
Datasets
MMSA-FET currently supports three MSA datasets: CMU-MOSI, CMU-MOSEI and CH-SIMS. The raw videos, feature files and label files can be downloaded from the following links.
- BaiduYun Disk
code:mfet
- Google Drive
You can add your own dataset as long as your dataset folder follow the structure described below.
Folder Structure & Label File
The datasets need to be organized in a specific folder structure for MMSA-FET to work properly on them.
-
The dataset files should be organized as follows:
. ├── DatasetName │ ├── label.csv // label file │ ├── Processed │ │ ├── feature_1.pkl // feature files │ │ ├── feature_2.pkl │ │ └── features.pkl │ ├── Raw │ │ ├── video_001 // video_id │ │ │ ├── 0001.mp4 // clip_id │ │ │ ├── 0002.mp4 │ │ │ ├── 0003.mp4 │ │ │ └── 0004.mp4 │ │ ├── video_002 // video_id │ │ │ ├── 0001.mp4 // clip_id │ │ │ ├── 0002.mp4 │ │ ├── video_003 ...
-
The
label.csv
file should contain columns in exact same order as the below example. The example corresponds to the above folder structure.video_id,clip_id,text,label,label_T,label_A,label_V,annotation,mode video_id ,clip_id ,text ,label ,label_T ,label_A ,label_V ,annotation ,mode video_001 ,0001 ,A LOT OF SAD PARTS ,-0.5 , , , ,Neutral ,train video_001 ,0002 ,THERE IS SAD PART ,-1.2 , , , ,Negative ,train video_001 ,0003 ,AND ITS A REALLY FUNNY ,1.8 , , , ,Positive ,train video_001 ,0004 ,BUT IT WAS REALLY REALLY AWESOME ,2.2 , , , ,Positive ,train video_002 ,0001 ,ANYHOW IT WAS REALLY GOOD ,2.4 , , , ,Positive ,train video_002 ,0002 ,I MEAN THEY DID A LITTLE BIT OF IT ,-1.0 , , , ,Negative ,train
Feature File
MMSA-FET generates feature pickle files which are compatible with MMSA. The feature file is organized as described in MMSA wiki