While not all runs are directly linked/documented, almost all the completed runs are here:

• Ablation final output/configs: https://huggingface.co/shisa-ai
• Training logs/configs: https://wandb.ai/augmxnt/shisa-v2

Current best model:

• ablation-53-rafathenev2.rp-shisa-v2-mistral-nemo-12b Current best llama3-8b
• ablation-66-a55.dpo.armorm-shisa-v2-llama-3.1-8b

This experiment tests whether pairwise training improves performance.

Start with our best shisa-v1.1 SFT ablation-11-rafathenev2 and try these four variations:

• seed=42 shuffle (should be almost the same as ablation 11, sanity check)
• seed=42 pairwise EN-JA
• seed=42 pairwise JA-EN
• seed=42 random pairwise

Sadly, SPIN simply does not replicate for us. It is worse that most of our DPO attempts, and worse than the base SFT model

• https://github.com/uclaml/SPIN

Why does this fail to replicate? Our theory is SPIN was tested on zephyr-7b-sft-full (Mistral 7B SFT'd on Ultrachat200k), a much weaker model. 50k was then taken from Ultrachat200k for generating new answers, with 50K/50K used for each iteration/n-1 iteration for stability.

In Abacus AI's Smaug DPOP paper, they point out one of DPO's failure modes is that if edit distances are low, then DPO can actually pick the worse answer and quality can go down so maybe this is related: https://arxiv.org/pdf/2402.13228

• https://github.com/abacusai/smaug/issues/2 - maybe also doesn't replicate well

Note, despite doing a test w/ an equal mix, it turns out that using a mix closer to DNO's (using 25% old data) seemed vs an equal amount seemed to be better.

Here's the topline summary: SPIN didn't seem to work at all on our test model. Each iteration is worse than the last, so something hinky is going on since the accuracy/margins go up normally for a DPO tune.

• The baseline takes 72K (same total number of rows that 3 iteration shows) to compare what the DPO does
• We pre-shuffle our dataset to make sure there is a mix of categories before we slice
• This recipe mixes some of the previous round in for stability
  • spin-01-iter0 (100%): ~20000K rows
  • spin-01-iter1 (100%) + spin-01-iter0 (25%)
  • spin-01-iter2 (100%) + spin-01-iter1 (25%) + spin-01-iter0 (5%)

The SPIN recipe uses 3 epochs of DPO. Is that the right number for each iteration? It seems, yes - overall, our output is better when we DPO more iterations:

As an interesting counterpoint, 1 epoch of the 75K baseline set does better with each epoch making the model perform worse:

Both iter1 runs are worse than iter0, but the 100% iter1 + 100% iter0 mix is worse than 100% iter1 and 25% iter0 data.

Basically no difference between armorm vs olmo2 preference mixes, except the latter takes a lot longer to train.

• Olmo preference mix takes 24h 42m https://wandb.ai/augmxnt/shisa-v2/runs/r9ycjoyt/overview
• The ArmoRM dpo set takes 1h 17m https://wandb.ai/augmxnt/shisa-v2/runs/vtuig65r/overview

The KTOTrainer does not appear to work with DeepSpeed-ZeRO 3

• https://github.com/huggingface/trl/issues/827

[ip-10-1-21-143:7]:[rank7]: Traceback (most recent call last):
[ip-10-1-21-143:7]:[rank7]:   File "<frozen runpy>", line 198, in _run_module_as_main
[ip-10-1-21-143:7]:[rank7]:   File "<frozen runpy>", line 88, in _run_code
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/cli/train.py", line 116, in <module>
[ip-10-1-21-143:7]:[rank7]:     fire.Fire(do_cli)
[ip-10-1-21-143:7]:[rank7]:   File "/fsx/ubuntu/miniforge3/envs/axolotl/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
[ip-10-1-21-143:7]:[rank7]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[ip-10-1-21-143:7]:[rank7]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx/ubuntu/miniforge3/envs/axolotl/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
[ip-10-1-21-143:7]:[rank7]:     component, remaining_args = _CallAndUpdateTrace(
[ip-10-1-21-143:7]:[rank7]:                                 ^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx/ubuntu/miniforge3/envs/axolotl/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[ip-10-1-21-143:7]:[rank7]:     component = fn(*varargs, **kwargs)
[ip-10-1-21-143:7]:[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/cli/train.py", line 90, in do_cli
[ip-10-1-21-143:7]:[rank7]:     return do_train(parsed_cfg, parsed_cli_args)
[ip-10-1-21-143:7]:[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/cli/train.py", line 46, in do_train
[ip-10-1-21-143:7]:[rank7]:     model, tokenizer, trainer = train(cfg=cfg, dataset_meta=dataset_meta)
[ip-10-1-21-143:7]:[rank7]:                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/train.py", line 483, in train
[ip-10-1-21-143:7]:[rank7]:     ) = setup_model_and_trainer(cfg, dataset_meta)
[ip-10-1-21-143:7]:[rank7]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/train.py", line 444, in setup_model_and_trainer
[ip-10-1-21-143:7]:[rank7]:     trainer = setup_trainer(
[ip-10-1-21-143:7]:[rank7]:               ^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/utils/trainer.py", line 615, in setup_trainer
[ip-10-1-21-143:7]:[rank7]:     return trainer_builder.build(total_num_steps)
[ip-10-1-21-143:7]:[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx2/axolotl/axolotl/src/axolotl/core/trainer_builder.py", line 1146, in build
[ip-10-1-21-143:7]:[rank7]:     dpo_trainer = trainer_cls(
[ip-10-1-21-143:7]:[rank7]:                   ^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx/ubuntu/miniforge3/envs/axolotl/lib/python3.12/site-packages/trl/trainer/kto_trainer.py", line 474, in __init__
[ip-10-1-21-143:7]:[rank7]:     self.ref_model = create_reference_model(model)
[ip-10-1-21-143:7]:[rank7]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ip-10-1-21-143:7]:[rank7]:   File "/fsx/ubuntu/miniforge3/envs/axolotl/lib/python3.12/site-packages/trl/models/modeling_base.py", line 622, in create_reference_model
[ip-10-1-21-143:7]:[rank7]:     raise ValueError(
[ip-10-1-21-143:7]:[rank7]: ValueError: DeepSpeed ZeRO-3 is enabled and is not compatible with `create_reference_model()`. Please instantiate your reference model directly with `AutoModelForCausalLM.from_pretrained()`.

Apparently TRL >0.13.0 is Broken w/ SimPO

• ablation-60-a55.dpo.enja-shisa-v2-llama-3.1-8b
• ablation-64-a55.simpo.enja-shisa-v2-llama-3.1-8b
• ablation-65-a55.simpo.armorm-shisa-v2-llama-3.1-8b
• See https://github.com/huggingface/trl/issues/2882 SimPO w/ https://github.com/princeton-nlp/SimPO https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm Currently broken in TRL for multiGPU?

OK, we can get this working on Axolotl 0.8.0 by downgrading trl from 0.15.1 (0.15.2 current) to 0.13.0

• related?
  • https://github.com/huggingface/trl/issues/2879
  • https://github.com/huggingface/trl/pull/2890
• ORPO also broken since 0.13.0
  • https://github.com/huggingface/trl/issues/2882

pip install trl==0.13.0

follow directions
make some changs to script imports

pip install wandb

[rank2]: Traceback (most recent call last):
[rank2]:   File "/fsx/ubuntu/meti/train/simpo/SimPO/scripts/run_simpo.py", line 316, in <module>
[rank2]:     main()
[rank2]:   File "/fsx/ubuntu/meti/train/simpo/SimPO/scripts/run_simpo.py", line 269, in main
[rank2]:     train_result = trainer.train(resume_from_checkpoint=checkpoint)
[rank2]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/fsx/ubuntu/miniforge3/envs/simpo/lib/python3.12/site-packages/transformers/trainer.py", line 2241, in train
[rank2]:     return inner_training_loop(
[rank2]:            ^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/fsx/ubuntu/miniforge3/envs/simpo/lib/python3.12/site-packages/transformers/trainer.py", line 2500, in _inner_training_loop
[rank2]:     batch_samples, num_items_in_batch = self.get_batch_samples(epoch_iterator, num_batches)
[rank2]:                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]:   File "/fsx/ubuntu/meti/train/simpo/SimPO/scripts/simpo_trainer.py", line 764, in get_batch_samples
[rank2]:     policy_output = model.generate(
[rank2]:                     ^^^^^^^^^^^^^^
[rank2]: AttributeError: 'generator' object has no attribute 'generate'

SimPO has was being a pain, but since they had a DPO set I decided to throw an ablation on to test. It gives a bit more of a boost than the EN/JA test set we were using.

• Preference set: https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm
• Notes:
  • https://github.com/princeton-nlp/SimPO
  • https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1

Adding the Yahoo! dataset has very little effect on Llama 3.1 8B

So we give this a try on Mistral Nemo 12B and get a rather disastrous result:

• brings down Shaberi scores
• improves mixeval
• dramatically lowers LiveBench
• slightly lowers llm-jp-eval
• dramatically lowers jp rp-bench
• lowers ifeval We do 2 eval runs just to confirm:

This initial test uses https://huggingface.co/datasets/shisa-ai/allenai_llama-3.1-tulu-3-405b-preference-mixture-filter-datecutoff_jpn - a 46.5K subset of AI2's 361K row Tulu 405B preference set: https://huggingface.co/datasets/allenai/llama-3.1-tulu-3-405b-preference-mixture that uses Tulu 405B to create Japanese translations.

We cannot use our dataset directly as there are mismatched rows that cause Axolotl to freak out. Instead, we have custom code that generates the clean prompt, chosen and rejected output:

• ablation-55-rafathenev2.rp.tl - current SFT mix
• ablation-58-a55.dpo.ultra - original EN chat results in a naive way (this is improper training)
• ablation-59-a55.dpo.en - EN chat set
• ablation-60-a55.dpo.enja - EN+JA chat sets shuffled
• ablation-61-a55.dpo.enjanot - EN+JA + and shuffles in variants using the EN/JA cross sets as rejected - the theory is this would help with replying in the correct language, but it trains double the data, so I guess we'll see what happens
• ablation-62-dpo.notseq - this takes ablation-60 (EN+JA) and then trains the EN/JA cross sets sequentially after

We train all of these for 1 epoch at a low 5e-7 LR w/ linear decay and we see steady improvement of margins and accuracy that show that the DPO is effective with our hyper-paramaters.

We don't get much benefit (slightly better, but really, within MOE):

• While it's good to have JA preference data, one issue with having Tulu 405B do translations is that it will even out quality to certain level (good/bad) - if we are using preference data, we really need to do our own native generations
• An additional factor is that translations are likely not the way to go vs generations from a quality perspective
• JA perf improved even with EN only preference data, which is a useful data point

Adding the translation result:

MODEL=ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-8b-lr8e6 OPENAI_URL=http://localhost:8000/v1 ./run_translation_bench.sh
vllm serve /fsx2/outputs/ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-8b-lr8e6 -tp 8 --gpu-memory-utilization 0.90 --num-scheduler-steps 20 --port 8000 --served-model-name ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-8b-lr8e6

EASY
---
9   ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-...   1.276373   171            299  2.652980e-01  7.818317e+00
9   ablation-55-rafathenev2.rp.tl-shisa-v2-llama-3...   0.862285   153            300  1.813070e-01  7.031378e+00

HARD
---
10  ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-...  0.147662   208            420  0.145712  5.368485
10  ablation-55-rafathenev2.rp.tl-shisa-v2-llama-3...  0.153531   208            420  0.153409  5.383076
ablation-46-rafathenev2.rp-shisa-v2-llama-3.1-... 5.368485

shisa-v1.1-nemo-12b shisa-v1.1-llama3.1-8b shisa-v1.1-gamma-7b shisa-v1.1-7b

This is to test if extra synthetic Japanese generation improves Japanese:

• Results are about the same except that JP-IFEval is better

This is just to confirm that a mathcode only mix improves our coding ability:

• Yes we get some improvements in EvalPlus
• However, LiveBench code and data analysis give us 0.0 so something seems wrong. Data format wrong?
  • Look at output.xlsx to see broken down scores
  • look at results for raw output

51 mathcode reasoning...

We should see a coding improvement if we use only the "good" English code?

• is the Tulumix just bad?
• It does not improve either our EN or JA when shuffled in. Every metric, including evalplus is worse.

Here's a comparison of just the EN vs our questionable JA

As a sanity check, we test whether it's better to do input masking: yes it is (this is w/ the flawed CMR mix):

Here is the just the cmrmix vs masked cmrmix

And here it is blended:

This experiment tests whether the addition of shisa-ai/shisa-v2-code-math-reasoning-sft-mix, a 192.4K EN/JA code/math/reasoning dataset improves our performance.

• ablation-11-rafathenev2 is shisa-v1.1-athenev2-filtered
• ablation-36-cmrmixshisa is the shisa-v2-code-math-reasoning-sft-mix alone
• ablation-37-rafathenev2.cmrmix shisa-v1.1-athenev2-filtered + shisa-v2-code-math-reasoning-sft-mix mixed
• ablation-38-rafathenv2.cmrmix.fixedlength - mixed, but 50% of each to approx same training length as ablation-11

• the JA crm-mix should improve coding but doesn't, so something very wrong! (sure enough, our translation model was eliding the conversion, no bueno)

This dataset is what we're using from allenai/tulu-3-sft-mixture with machine translation to Japanese.

https://huggingface.co/datasets/shisa-ai/shisa-v2-code-math-reasoning-sft-mix/viewer/default/train?p=2&sql=--+The+SQL+console+is+powered+by+DuckDB+WASM+and+runs+entirely+in+the+browser.%0A--+Get+started+by+typing+a+query+or+selecting+a+view+from+the+options+below.%0ASELECT+DISTINCT%28SOURCE%29+FROM+train%3B&views%5B%5D=train
-- The SQL console is powered by DuckDB WASM and runs entirely in the browser.
-- Get started by typing a query or selecting a view from the options below.
SELECT DISTINCT(SOURCE) FROM train;

ai2-adapt-dev/no_robots_converted
allenai/tulu-3-sft-personas-math-grade
ai2-adapt-dev/tulu_v3.9_open_math_2_gsm8k_50k
ai2-adapt-dev/evol_codealpaca_heval_decontaminated
ai2-adapt-dev/flan_v2_converted
ai2-adapt-dev/personahub_code_v2_34999
ai2-adapt-dev/numinamath_tir_math_decontaminated

In axolotl, the formatting is Tulu's, and here's how to train:

  - path: shisa-ai/shisa-v2-code-math-reasoning-sft-mix
    type: chat_template
    field_messages: conversations
    message_property_mappings:
      role: role
      content: content
    roles:
      system:
        - system
      assistant:
        - gpt
        - model
        - assistant
      user:
        - human
        - user
    roles_to_train: ["system", "input", "assistant"]

For ablation-37 we add split: train[:50%] to the config (see: https://github.com/axolotl-ai-cloud/axolotl/discussions/2280)

• ablation-01
• ablation-02
• ablation-03
• ablation-04

Confirmation that using the Liger Kernels improves training efficiency and doesn't negatively affect performance.

This is a sanity check using basically the exact same config (ShareGPT config has changed to updated syntax). The only thing that has changed is updated versions of the software (May 2024 vs Feb 2025).

• shisa-v1-llama3-8b
  • https://huggingface.co/shisa-ai/shisa-v1-llama3-8b
  • https://wandb.ai/augmxnt/shisa-v2/runs/i2y0l39o
• ablation-00-baseline-shisa-v2-llama3-8b-lr-8e6
  • https://huggingface.co/shisa-ai/ablation-00-baseline-shisa-v2-llama3-8b-lr8e6
  • https://wandb.ai/augmxnt/shisa-v2/runs/4f8nwu8e

Trained from:

• Llama 3 8B Instruct
  • https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct shisa-v1 og dataset
  • https://huggingface.co/datasets/augmxnt/ultra-orca-boros-en-ja-v1

On almost every metric, our new ablation performs better using the same hyper-parameters and datasets. I guess there were a lot of under the hood bugs/improvements made between PyTorch, Transformers, TRL, and Axolotl...

• Overall score is +17.5%
• see https://axolotl-ai-cloud.github.io/axolotl/docs/multipack.html and https://github.com/axolotl-ai-cloud/axolotl/issues/1375

https://github.com/axolotl-ai-cloud/axolotl/blob/main/src/axolotl/utils/data/sft.py https://axolotl-ai-cloud.github.io/axolotl/docs/dataset_preprocessing.html https://github.com/axolotl-ai-cloud/axolotl/issues/1508 https://github.com/axolotl-ai-cloud/axolotl/issues/735 https://github.com/axolotl-ai-cloud/axolotl/discussions/2280 https://github.com/axolotl-ai-cloud/axolotl/issues/735