Dataset and Structure - thuiar/MMSA-FET GitHub Wiki

Datasets

MMSA-FET currently supports three MSA datasets: CMU-MOSI, CMU-MOSEI and CH-SIMS. The raw videos, feature files and label files can be downloaded from the following links.

You can add your own dataset as long as your dataset folder follow the structure described below.

Folder Structure & Label File

The datasets need to be organized in a specific folder structure for MMSA-FET to work properly on them.

  • The dataset files should be organized as follows:

    .
    ├── DatasetName
    │   ├── label.csv           // label file
    │   ├── Processed
    │   │   ├── feature_1.pkl   // feature files
    │   │   ├── feature_2.pkl
    │   │   └── features.pkl
    │   ├── Raw
    │   │   ├── video_001       // video_id
    │   │   │   ├── 0001.mp4    // clip_id
    │   │   │   ├── 0002.mp4
    │   │   │   ├── 0003.mp4
    │   │   │   └── 0004.mp4
    │   │   ├── video_002       // video_id
    │   │   │   ├── 0001.mp4    // clip_id
    │   │   │   ├── 0002.mp4
    │   │   ├── video_003
    ...
    
  • The label.csv file should contain columns in exact same order as the below example. The example corresponds to the above folder structure.

    video_id,clip_id,text,label,label_T,label_A,label_V,annotation,mode
    video_id  ,clip_id ,text                               ,label ,label_T ,label_A ,label_V ,annotation ,mode
    video_001 ,0001    ,A LOT OF SAD PARTS                 ,-0.5  ,        ,        ,        ,Neutral    ,train
    video_001 ,0002    ,THERE IS SAD PART                  ,-1.2  ,        ,        ,        ,Negative   ,train
    video_001 ,0003    ,AND ITS A REALLY FUNNY             ,1.8   ,        ,        ,        ,Positive   ,train
    video_001 ,0004    ,BUT IT WAS REALLY REALLY AWESOME   ,2.2   ,        ,        ,        ,Positive   ,train
    video_002 ,0001    ,ANYHOW IT WAS REALLY GOOD          ,2.4   ,        ,        ,        ,Positive   ,train
    video_002 ,0002    ,I MEAN THEY DID A LITTLE BIT OF IT ,-1.0  ,        ,        ,        ,Negative   ,train
    

Feature File

MMSA-FET generates feature pickle files which are compatible with MMSA. The feature file is organized as described in MMSA wiki