Downstream Task Instructions - andi611/Mockingjay-Speech-Representation GitHub Wiki
Task 1 - Training and Testing of the Phone Classification Task
Run the following command to train a phone classifier:
# using spectrogram as baseline
python3 runner_mockingjay.py --train_phone
# using mockingjay representations
python3 runner_mockingjay.py --train_phone --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_phone --run_apc
Run the following command to test representations using the phone classifier:
# testing spectrogram as baseline
python3 runner_mockingjay.py --test_phone
# testing mockingjay representations
python3 runner_mockingjay.py --test_phone --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_phone_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_phone --run_apc
To link and compare with the CPC work, we can train with the identical aligned phone labels and train/test split as in the CPC paper. Make sure that you have followed the instructions in the downstream task preprocessing wiki page, and the Kaldi feature extraction wiki page. Then edit the following attributes in your .yaml config files, and use the following commands:
# mockingjay_libri_fmllrBase.yaml
data_path: 'data/libri_fmllr_cmvn' # Features extracted by Kaldi (http://www.kaldi-asr.org/downloads/build/6/trunk/egs/librispeech/)
phone_path: 'data/cpc_phone'
train_set: ['train-clean-100']
# using spectrogram as baseline
python3 runner_mockingjay.py --train_cpc_phone
# using mockingjay representations
python3 runner_mockingjay.py --train_cpc_phone --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_cpc_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_cpc_phone --run_apc
Task 2 - Training and Testing of the Sentimental Classification Task
Run the following command to train a sentiment classifier:
# using spectrogram as baseline
python3 runner_mockingjay.py --train_sentiment
# using mockingjay representations
python3 runner_mockingjay.py --train_sentiment --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_sentiment --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_sentiment --run_apc
Run the following command to test representations using the phone classifier:
# testing spectrogram as baseline
python3 runner_mockingjay.py --test_sentiment
# testing mockingjay representations
python3 runner_mockingjay.py --test_sentiment --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_sentiment --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_sentiment_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_sentiment --run_apc
Task 3 - Training and Testing of the Speaker Verification Task
Run the following command to train a sentiment classifier:
# using spectrogram as baseline
python3 runner_mockingjay.py --train_speaker
# using mockingjay representations
python3 runner_mockingjay.py --train_speaker --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_speaker --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_speaker --run_apc
Run the following command to test representations using the phone classifier:
# testing spectrogram as baseline
python3 runner_mockingjay.py --test_speaker
# testing mockingjay representations
python3 runner_mockingjay.py --test_speaker --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_speaker --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_speaker_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_speaker --run_apc