From 4f403ea8994ee8785aa73957c827938e74cf0fe3 Mon Sep 17 00:00:00 2001 From: Daniel Erenrich Date: Fri, 11 Feb 2022 16:51:30 -0500 Subject: [PATCH] Fix grammar in tokenizer_summary (#15614) "to make ensure" is redundant. --- docs/source/tokenizer_summary.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/tokenizer_summary.mdx b/docs/source/tokenizer_summary.mdx index 401c620d00..7827839030 100644 --- a/docs/source/tokenizer_summary.mdx +++ b/docs/source/tokenizer_summary.mdx @@ -219,7 +219,7 @@ equivalent to finding the symbol pair, whose probability divided by the probabil its second symbol is the greatest among all symbol pairs. *E.g.* `"u"`, followed by `"g"` would have only been merged if the probability of `"ug"` divided by `"u"`, `"g"` would have been greater than for any other symbol pair. Intuitively, WordPiece is slightly different to BPE in that it evaluates what it _loses_ by merging two symbols -to make ensure it's _worth it_. +to ensure it's _worth it_.