SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Update chat template docs and bump Jinja version (#31455)

Browse Source 
* Update chat template docs

* Minor bug in the version check

* Update docs/source/en/chat_templating.md

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Update docs/source/en/chat_templating.md

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Update docs/source/en/chat_templating.md

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Replace backticks with bolding because the doc builder was trying to parse them

* Replace backticks with bolding because the doc builder was trying to parse them

* Replace backticks with bolding because the doc builder was trying to parse them

* More cleanups to avoid upsetting the doc builder

* Add one more tip at the end

---------

Co-authored-by: Joshua Lochner <admin@xenova.com>
This commit is contained in:
Matt
2024-06-18 14:16:30 +01:00
committed by GitHub
parent 28316d0e8b
commit 6e56b83453
3 changed files with 90 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files

140
docs/source/en/chat_templating.md
View File

@@ -573,23 +573,21 @@ default template for that model class is used instead. Let's take a look at the
"{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ ' ' }}{% endif %}{% endfor %}{{ eos_token }}" "{% for message in messages %}{% if message['role'] == 'user' %}{{ ' ' }}{% endif %}{{ message['content'] }}{% if not loop.last %}{{ ' ' }}{% endif %}{% endfor %}{{ eos_token }}"
``` ```
That's kind of intimidating. Let's add some newlines and indentation to make it more readable. Note that the first That's kind of intimidating. Let's clean it up a little to make it more readable. In the process, though, we also make
newline after each block as well as any preceding whitespace before a block are ignored by default, using the sure that the newlines and indentation we add don't end up being included in the template output - see the tip on
Jinja `trim_blocks` and `lstrip_blocks` flags. However, be cautious - although leading whitespace on each [trimming whitespace](#trimming-whitespace) below!
line is stripped, spaces between blocks on the same line are not. We strongly recommend checking that your template
isn't printing extra spaces where it shouldn't be!
``` ```
{% for message in messages %} {%- for message in messages %}
{% if message['role'] == 'user' %} {%- if message['role'] == 'user' %}
{{ ' ' }} {{- ' ' }}
{% endif %} {%- endif %}
{{ message['content'] }} {{- message['content'] }}
{% if not loop.last %} {%- if not loop.last %}
{{ ' ' }} {{- ' ' }}
{% endif %} {%- endif %}
{% endfor %} {%- endfor %}
{{ eos_token }} {{- eos_token }}
``` ```
If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/). If you've never seen one of these before, this is a [Jinja template](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -618,15 +616,15 @@ similarly to the way LLaMA formats them (note that the real LLaMA template inclu
messages and slightly different system message handling in general - don't use this one in your actual code!) messages and slightly different system message handling in general - don't use this one in your actual code!)
``` ```
{% for message in messages %} {%- for message in messages %}
{% if message['role'] == 'user' %} {%- if message['role'] == 'user' %}
{{ bos_token + '[INST] ' + message['content'] + ' [/INST]' }} {{- bos_token + '[INST] ' + message['content'] + ' [/INST]' }}
{% elif message['role'] == 'system' %} {%- elif message['role'] == 'system' %}
{{ '<<SYS>>\\n' + message['content'] + '\\n<</SYS>>\\n\\n' }} {{- '<<SYS>>\\n' + message['content'] + '\\n<</SYS>>\\n\\n' }}
{% elif message['role'] == 'assistant' %} {%- elif message['role'] == 'assistant' %}
{{ ' ' + message['content'] + ' ' + eos_token }} {{- ' ' + message['content'] + ' ' + eos_token }}
{% endif %} {%- endif %}
{% endfor %} {%- endfor %}
``` ```
Hopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based Hopefully if you stare at this for a little bit you can see what this template is doing - it adds specific tokens based
@@ -642,15 +640,15 @@ existing template from another model and simply edit it for your needs! For exam
above and add "[ASST]" and "[/ASST]" to assistant messages: above and add "[ASST]" and "[/ASST]" to assistant messages:
``` ```
{% for message in messages %} {%- for message in messages %}
{% if message['role'] == 'user' %} {%- if message['role'] == 'user' %}
{{ bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }} {{- bos_token + '[INST] ' + message['content'].strip() + ' [/INST]' }}
{% elif message['role'] == 'system' %} {%- elif message['role'] == 'system' %}
{{ '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }} {{- '<<SYS>>\\n' + message['content'].strip() + '\\n<</SYS>>\\n\\n' }}
{% elif message['role'] == 'assistant' %} {%- elif message['role'] == 'assistant' %}
{{ '[ASST] ' + message['content'] + ' [/ASST]' + eos_token }} {{- '[ASST] ' + message['content'] + ' [/ASST]' + eos_token }}
{% endif %} {%- endif %}
{% endfor %} {%- endfor %}
``` ```
Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will Now, simply set the `tokenizer.chat_template` attribute. Next time you use [`~PreTrainedTokenizer.apply_chat_template`], it will
@@ -726,9 +724,9 @@ input formats. One popular choice is the `ChatML` format, and this is a good, fl
It looks like this: It looks like this:
``` ```
{% for message in messages %} {%- for message in messages %}
{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}} {{- '<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n' }}
{% endfor %} {%- endfor %}
``` ```
If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes If you like this one, here it is in one-liner form, ready to copy into your code. The one-liner also includes
@@ -776,21 +774,43 @@ it's time to put an end to them!
If you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first If you're unfamiliar with Jinja, we generally find that the easiest way to write a chat template is to first
write a short Python script that formats messages the way you want, and then convert that script into a template. write a short Python script that formats messages the way you want, and then convert that script into a template.
Remember that the template handler will receive the conversation history as a variable called `messages`. Each Remember that the template handler will receive the conversation history as a variable called `messages`.
message is a dictionary with two keys, `role` and `content`. You will be able to access `messages` in your template You will be able to access `messages` in your template just like you can in Python, which means you can loop over
just like you can in Python, which means you can loop over it with `{% for message in messages %}` or access it with `{% for message in messages %}` or access individual messages with `{{ messages[0] }}`, for example.
individual messages with, for example, `{{ messages[0] }}`.
You can also use the following tips to convert your code to Jinja: You can also use the following tips to convert your code to Jinja:
### Trimming whitespace
By default, Jinja will print any whitespace that comes before or after a block. This can be a problem for chat
templates, which generally want to be very precise with whitespace! To avoid this, we strongly recommend writing
your templates like this:
```
{%- for message in messages %}
{{- message['role'] + message['content'] }}
{%- endfor %}
```
rather than like this:
```
{% for message in messages %}
{{ message['role'] + message['content'] }}
{% endfor %}
```
Adding `-` will strip any whitespace that comes before the block. The second example looks innocent, but the newline
and indentation may end up being included in the output, which is probably not what you want!
### For loops ### For loops
For loops in Jinja look like this: For loops in Jinja look like this:
``` ```
{% for message in messages %} {%- for message in messages %}
{{ message['content'] }} {{- message['content'] }}
{% endfor %} {%- endfor %}
``` ```
Note that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like Note that whatever's inside the {{ expression block }} will be printed to the output. You can use operators like
@@ -801,9 +821,9 @@ Note that whatever's inside the {{ expression block }} will be printed to the ou
If statements in Jinja look like this: If statements in Jinja look like this:
``` ```
{% if message['role'] == 'user' %} {%- if message['role'] == 'user' %}
{{ message['content'] }} {{- message['content'] }}
{% endif %} {%- endif %}
``` ```
Note how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you Note how where Python uses whitespace to mark the beginnings and ends of `for` and `if` blocks, Jinja requires you
@@ -819,14 +839,26 @@ conversation. Here's an example that puts these ideas together to add a generati
conversation if add_generation_prompt is `True`: conversation if add_generation_prompt is `True`:
``` ```
{% if loop.last and add_generation_prompt %} {%- if loop.last and add_generation_prompt %}
{{ bos_token + 'Assistant:\n' }} {{- bos_token + 'Assistant:\n' }}
{% endif %} {%- endif %}
``` ```
### Notes on whitespace ### Compatibility with non-Python Jinja
As much as possible, we've tried to get Jinja to ignore whitespace outside of {{ expressions }}. However, be aware There are multiple implementations of Jinja in various languages. They generally have the same syntax,
that Jinja is a general-purpose templating engine, and it may treat whitespace between blocks on the same line but a key difference is that when you're writing a template in Python you can use Python methods, such as
as significant and print it to the output. We **strongly** recommend checking that your template isn't printing extra `.lower()` on strings or `.items()` on dicts. This will break if someone tries to use your template on a non-Python
spaces where it shouldn't be before you upload it! implementation of Jinja. Non-Python implementations are particularly common in deployment environments, where JS
and Rust are very popular.
Don't panic, though! There are a few easy changes you can make to your templates to ensure they're compatible across
all implementations of Jinja:
- Replace Python methods with Jinja filters. These usually have the same name, for example `string.lower()` becomes
`string|lower`, and `dict.items()` becomes `dict|items`. One notable change is that `string.strip()` becomes `string|trim`.
See the [list of built-in filters](https://jinja.palletsprojects.com/en/3.1.x/templates/#builtin-filters)
in the Jinja documentation for more.
- Replace `True`, `False` and `None`, which are Python-specific, with `true`, `false` and `none`.
- Directly rendering a dict or list may give different results in other implementations (for example, string entries
might change from single-quoted to double-quoted). Adding the `tojson` filter can help to ensure consistency here.

4
src/transformers/agents/tools.py
View File

@@ -441,8 +441,8 @@ def compile_jinja_template(template):
except ImportError: except ImportError:
raise ImportError("template requires jinja2 to be installed.") raise ImportError("template requires jinja2 to be installed.")
if version.parse(jinja2.__version__) <= version.parse("3.0.0"): if version.parse(jinja2.__version__) < version.parse("3.1.0"):
raise ImportError("template requires jinja2>=3.0.0 to be installed. Your version is " f"{jinja2.__version__}.") raise ImportError("template requires jinja2>=3.1.0 to be installed. Your version is " f"{jinja2.__version__}.")
def raise_exception(message): def raise_exception(message):
raise TemplateError(message) raise TemplateError(message)

4
src/transformers/tokenization_utils_base.py
View File

@@ -1890,9 +1890,9 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
except ImportError: except ImportError:
raise ImportError("apply_chat_template requires jinja2 to be installed.") raise ImportError("apply_chat_template requires jinja2 to be installed.")
if version.parse(jinja2.__version__) < version.parse("3.0.0"): if version.parse(jinja2.__version__) < version.parse("3.1.0"):
raise ImportError( raise ImportError(
"apply_chat_template requires jinja2>=3.0.0 to be installed. Your version is " f"{jinja2.__version__}." "apply_chat_template requires jinja2>=3.1.0 to be installed. Your version is " f"{jinja2.__version__}."
) )
def raise_exception(message): def raise_exception(message):