From 9625924c605685c76fcbe4161ed90141a4989370 Mon Sep 17 00:00:00 2001 From: bofeng huang Date: Tue, 15 Nov 2022 01:18:13 +0100 Subject: [PATCH] Update tokenizer_summary.mdx (#20135) --- docs/source/en/tokenizer_summary.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/en/tokenizer_summary.mdx b/docs/source/en/tokenizer_summary.mdx index 7827839030..942fe27906 100644 --- a/docs/source/en/tokenizer_summary.mdx +++ b/docs/source/en/tokenizer_summary.mdx @@ -86,7 +86,7 @@ representation for the letter `"t"` is much harder than learning a context-indep both worlds, transformers models use a hybrid between word-level and character-level tokenization called **subword** tokenization. -### Subword tokenization +## Subword tokenization @@ -133,7 +133,7 @@ on. -## Byte-Pair Encoding (BPE) +### Byte-Pair Encoding (BPE) Byte-Pair Encoding (BPE) was introduced in [Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2015)](https://arxiv.org/abs/1508.07909). BPE relies on a pre-tokenizer that splits the training data into @@ -194,7 +194,7 @@ As mentioned earlier, the vocabulary size, *i.e.* the base vocabulary size + the to choose. For instance [GPT](model_doc/gpt) has a vocabulary size of 40,478 since they have 478 base characters and chose to stop training after 40,000 merges. -### Byte-level BPE +#### Byte-level BPE A base vocabulary that includes all possible base characters can be quite large if *e.g.* all unicode characters are considered as base characters. To have a better base vocabulary, [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) uses bytes @@ -206,7 +206,7 @@ with 50,000 merges. -#### WordPiece +### WordPiece WordPiece is the subword tokenization algorithm used for [BERT](model_doc/bert), [DistilBERT](model_doc/distilbert), and [Electra](model_doc/electra). The algorithm was outlined in [Japanese and Korean Voice Search (Schuster et al., 2012)](https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf) and is very similar to @@ -223,7 +223,7 @@ to ensure it's _worth it_. -#### Unigram +### Unigram Unigram is a subword tokenization algorithm introduced in [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018)](https://arxiv.org/pdf/1804.10959.pdf). In contrast to BPE or @@ -260,7 +260,7 @@ $$\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right ) -#### SentencePiece +### SentencePiece All tokenization algorithms described so far have the same problem: It is assumed that the input text uses spaces to separate words. However, not all languages use spaces to separate words. One possible solution is to use language