SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Fix grammar in tokenizer_summary (#15614)

Browse Source 
"to make ensure" is redundant.
This commit is contained in:
Daniel Erenrich
2022-02-11 16:51:30 -05:00
committed by GitHub
parent 7a32e4722f
commit 4f403ea899
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/tokenizer_summary.mdx
View File

@@ -219,7 +219,7 @@ equivalent to finding the symbol pair, whose probability divided by the probabil
its second symbol is the greatest among all symbol pairs. *E.g.* `"u"`, followed by `"g"` would have only been its second symbol is the greatest among all symbol pairs. *E.g.* `"u"`, followed by `"g"` would have only been
merged if the probability of `"ug"` divided by `"u"`, `"g"` would have been greater than for any other symbol merged if the probability of `"ug"` divided by `"u"`, `"g"` would have been greater than for any other symbol
pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols
to make ensure it's _worth it_. to ensure it's _worth it_.
<a id='unigram'></a> <a id='unigram'></a>