SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Added CamembertForQuestionAnswering (#2746)

Browse Source 
* Added CamembertForQuestionAnswering

* fixed camembert tokenizer case
This commit is contained in:
maximeilluin
2020-02-21 18:01:02 +01:00
committed by GitHub
parent 5211d333bb
commit c749a543fa
4 changed files with 30 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/data/processors/squad.py
View File

@@ -123,7 +123,7 @@ def squad_convert_example_to_features(example, max_seq_length, doc_stride, max_q
truncated_query = tokenizer.encode(example.question_text, add_special_tokens=False, max_length=max_query_length)
sequence_added_tokens = (
tokenizer.max_len - tokenizer.max_len_single_sentence + 1
if "roberta" in str(type(tokenizer))
if "roberta" in str(type(tokenizer)) or "camembert" in str(type(tokenizer))
else tokenizer.max_len - tokenizer.max_len_single_sentence
)
sequence_pair_added_tokens = tokenizer.max_len - tokenizer.max_len_sentences_pair