Chat Templates - shisa-ai/shisa-v2 GitHub Wiki

Chat Templates can be tricky. Here's some notes:

HF rolled out chat_template last year which uses Jinja2 templates basically:

This chat_templates repo tries to gather/give readable implementations of different chat template formats:

Two Jinja live dev envs:

They might not like raise_exception. You can use something like this for the variables:

bos_token: "[B]"
eos_token: "[E]"
messages:
  - role: system
    content: This is the system prompt.
  - role: user
    content: This is the first user input.
  - role: assistant
    content: This is the first assistant response.
  - role: user
    content: This is the second user input.

NOTE: You usually won't want to prepend spaces to special tokens, these should be added by the tokenizer and aren't reflected in the Jinja previews.

Llama 3

Axolotl puts in an extra <|eot_id|> in their chat template. Fixed here: https://github.com/OpenAccess-AI-Collective/axolotl/pull/1635

Mistral

Mistral has a weird inst template.

Here's how the Mixtral-8x7B-Instruct-v0.1 model card describes it:

<s> [INST] Instruction [/INST] Model answer</s> [INST] Follow-up instruction [/INST]

and from their tokenizer_config.json:

"chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}"

Axolotl uses exactly this format.

Here's chat_template repo's version: https://github.com/chujiezheng/chat_templates/blob/main/chat_templates/mistral-instruct.jinja

It adds support for a system prompt:

{% if messages[0]['role'] == 'system' %}
    {% set loop_messages = messages[1:] %}
    {% set system_message = messages[0]['content'].strip() + '\n\n' %}
{% else %}
    {% set loop_messages = messages %}
    {% set system_message = '' %}
{% endif %}

{{ bos_token }}
{% for message in loop_messages %}
    {% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}
        {{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}
    {% endif %}
    
    {% if loop.index0 == 0 %}
        {% set content = system_message + message['content'] %}
    {% else %}
        {% set content = message['content'] %}
    {% endif %}
    
    {% if message['role'] == 'user' %}
        {{ '[INST] ' + content.strip() + ' [/INST]' }}
    {% elif message['role'] == 'assistant' %}
        {{ ' ' + content.strip() + eos_token }}
    {% endif %}
{% endfor %}

Since I wanted system prompts, here's what I ended up writing for shisa-swallowmx-13a47b-v1:

"chat_template": "{% if messages[0]['role'] == 'system' %}{% set offset = 1 %}{% else %}{% set offset = 0 %}{% endif %}{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == offset) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% elif message['role'] == 'system' %}{{ message['content'] }}{% endif %}{% endfor %}",
⚠️ **GitHub.com Fallback** ⚠️