HuggingFace_transformer

Files

Sylvain Gugger dc42e770b8 Easily train a new fast tokenizer from a given one (#12361 )

* [WIP] Easily train a new fast tokenizer from a given one

* Fix test

* Roll out to other tokenizers and add tests

* Fix bug with unk id and add emoji to test

* Really use something different in test

* Implement special tokens map

* Map special tokens in the Transformers tokenizers

* Fix test

* Make test more robust

* Fix test for BPE

* More robust map and test

Co-authored-by SaulLu

* Test file

* Stronger tests

Co-authored-by: SaulLu <lucilesaul.com@gmail.com>

* Map unk token for Wordpiece and address review comment

* Fix lowercase test and address review comment

* Fix all tests

* Simplify test

* Fix tests for realsies

* Easily train a new fast tokenizer from a given one - tackle the special tokens format (str or AddedToken) (#12420)

* Propose change in tests regarding lower case

* add new test for special tokens types

* put back the test part about decoding

* add feature: the AddedToken is re-build with the different mapped content

* Address review comment: simplify AddedToken building

Co-authored-by: sgugger <sylvain.gugger@gmail.com>

* Update src/transformers/tokenization_utils_fast.py

Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: SaulLu <lucilesaul.com@gmail.com>
Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

2021-06-29 15:00:08 -04:00

deepspeed

[trainer] 2 bug fixes and a rename (#12309 )

2021-06-22 11:13:23 -07:00

extended

[trainer] 2 bug fixes and a rename (#12309 )

2021-06-22 11:13:23 -07:00

fixtures

Add DETR (#11653 )

2021-06-09 11:51:13 -04:00

sagemaker

[Examples] Added context manager to datasets map (#12367 )

2021-06-28 09:14:00 -07:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

conftest.py

Trainer push to hub (#11328 )

2021-04-23 09:17:37 -04:00

test_activations_tf.py

2020-12-07 18:36:34 -05:00

test_activations.py

Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027 )

2021-06-08 13:58:38 +01:00

test_benchmark_tf.py

2020-12-07 18:36:34 -05:00

test_benchmark.py

2020-12-07 18:36:34 -05:00

test_cli.py

2020-12-07 18:36:34 -05:00

test_configuration_auto.py

Reorganize repo (#8580 )

2020-11-16 21:43:42 -05:00

test_configuration_common.py

Clean push to hub API (#12187 )

2021-06-23 10:11:19 -04:00

test_data_collator.py

Run mlm pad to multiple for fp16 (#11128 )

2021-04-08 16:12:49 -04:00

test_doc_samples.py

Fix ignore list behavior in doctests (#8213 )

2020-11-02 08:47:37 -05:00

test_feature_extraction_auto.py

Add the ImageClassificationPipeline (#11598 )

2021-05-07 08:08:40 -04:00

test_feature_extraction_clip.py

CLIP (#11445 )

2021-05-12 13:48:15 +05:30

test_feature_extraction_common.py

Add DETR (#11653 )

2021-06-09 11:51:13 -04:00

test_feature_extraction_deit.py

Add DETR (#11653 )

2021-06-09 11:51:13 -04:00

test_feature_extraction_detr.py

Rename detr targets to labels (#12280 )

2021-06-29 03:07:46 -04:00

test_feature_extraction_speech_to_text.py

Auto feature extractor (#11097 )

2021-04-06 19:20:08 -04:00

test_feature_extraction_vit.py

Add DETR (#11653 )

2021-06-09 11:51:13 -04:00

test_feature_extraction_wav2vec2.py

W2v2 test require torch (#10665 )

2021-03-11 12:56:12 -05:00

test_file_utils.py

Patches the full import failure and adds a test (#10750 )

2021-03-16 15:37:52 -04:00

test_flax_auto.py

[ci, flax] non-existing models are unlikely to pass tests (#10409 )

2021-02-26 12:35:36 +03:00

test_generation_beam_search.py

Remove max length beam scorer (#11378 )

2021-04-27 00:28:40 +02:00

test_generation_flax_logits_process.py

[Flax] Add Beam Search (#12131 )

2021-06-16 09:43:54 +01:00

test_generation_flax_utils.py

[Flax] Add Beam Search (#12131 )

2021-06-16 09:43:54 +01:00

test_generation_logits_process.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_generation_stopping_criteria.py

Adding new argument max_new_tokens for generate. (#11476 )

2021-05-27 14:22:58 +02:00

test_generation_utils.py

Fix head masking generate tests (#12110 )

2021-06-11 04:04:07 -04:00

test_hf_api.py

Trainer push to hub (#11328 )

2021-04-23 09:17:37 -04:00

test_hf_argparser.py

Adding required flags to non-default arguments in hf_argparser (#10688 )

2021-03-15 09:27:55 -04:00

test_image_utils.py

Add center_crop to ImageFeatureExtractoMixin (#11066 )

2021-04-05 15:28:51 -04:00

test_logging.py

fix: The 'warn' method is deprecated (#11105 )

2021-04-07 09:20:06 -04:00

test_model_card.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_model_output.py

Add tests and fix various bugs in ModelOutput (#7073 )

2020-09-11 12:01:33 -04:00

test_modeling_albert.py

A cleaner and more scalable implementation of symbolic tracing (#11763 )

2021-05-20 18:02:29 +02:00

test_modeling_auto.py

Add support for multiple models for one config in auto classes (#11150 )

2021-04-08 18:41:36 -04:00

test_modeling_bart.py

Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027 )

2021-06-08 13:58:38 +01:00

test_modeling_bert_generation.py

Replace error by warning when loading an architecture in another (#11207 )

2021-04-13 10:33:52 -04:00

test_modeling_bert.py

Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475 )

2021-05-14 20:57:30 +02:00

test_modeling_big_bird.py

Add multi-class, multi-label and regression to transformers (#11012 )

2021-05-04 02:23:40 -04:00

test_modeling_bigbird_pegasus.py

Add some tests to the slow suite #11860

2021-05-25 04:06:06 -04:00

test_modeling_blenderbot_small.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_blenderbot.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_bort.py

ADD BORT (#9813 )

2021-01-27 21:25:11 +03:00

test_modeling_camembert.py

2020-12-07 18:36:34 -05:00

test_modeling_clip.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_common.py

[models] respect dtype of the model when instantiating it (#12316 )

2021-06-28 20:11:21 -07:00

test_modeling_convbert.py

Add multi-class, multi-label and regression to transformers (#11012 )

2021-05-04 02:23:40 -04:00

test_modeling_ctrl.py

Ctrl for sequence classification (#8812 )

2020-12-01 09:49:27 +01:00

test_modeling_deberta_v2.py

Fix integration slow tests (#10670 )

2021-03-11 13:43:53 -05:00

test_modeling_deberta.py

Fix integration slow tests (#10670 )

2021-03-11 13:43:53 -05:00

test_modeling_deit.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_detr.py

Add DETR (#11653 )

2021-06-09 11:51:13 -04:00

test_modeling_distilbert.py

A cleaner and more scalable implementation of symbolic tracing (#11763 )

2021-05-20 18:02:29 +02:00

test_modeling_dpr.py

Fix slow dpr test (#10059 )

2021-02-08 04:43:25 -05:00

test_modeling_electra.py

Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475 )

2021-05-14 20:57:30 +02:00

test_modeling_encoder_decoder.py

EncoderDecoderConfigs should not create new objects (#11300 )

2021-04-25 11:45:46 +02:00

test_modeling_flaubert.py

Integration test for FlauBert (#10022 )

2021-02-08 04:36:50 -05:00

test_modeling_flax_bart.py

Flax T5 (#12150 )

2021-06-23 13:13:32 +01:00

test_modeling_flax_bert.py

Add support for multiple models for one config in auto classes (#11150 )

2021-04-08 18:41:36 -04:00

test_modeling_flax_big_bird.py

Flax Big Bird (#11967 )

2021-06-14 20:01:03 +01:00

test_modeling_flax_clip.py

[Flax] [WIP] allow loading head model with base model weights (#12255 )

2021-06-21 15:56:42 +01:00

test_modeling_flax_common.py

Clean push to hub API (#12187 )

2021-06-23 10:11:19 -04:00

test_modeling_flax_electra.py

[Flax] Add Electra models (#11426 )

2021-05-04 20:56:09 +02:00

test_modeling_flax_gpt2.py

Flax Generate (#11777 )

2021-05-27 00:18:17 +01:00

test_modeling_flax_roberta.py

[FlaxRoberta] Add FlaxRobertaModels & adapt run_mlm_flax.py (#11470 )

2021-05-04 19:57:59 +02:00

test_modeling_flax_t5.py

Flax T5 (#12150 )

2021-06-23 13:13:32 +01:00

test_modeling_flax_vit.py

Flax Big Bird (#11967 )

2021-06-14 20:01:03 +01:00

test_modeling_fsmt.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_funnel.py

Pytorch - Lazy initialization of models (#11471 )

2021-05-05 17:22:20 +02:00

test_modeling_gpt2.py

Added Sequence Classification class in GPTNeo (#11906 )

2021-05-28 06:27:02 -04:00

test_modeling_gpt_neo.py

Added Sequence Classification class in GPTNeo (#11906 )

2021-05-28 06:27:02 -04:00

test_modeling_hubert.py

Add out of vocabulary error to ASR models (#12288 )

2021-06-29 08:57:46 +01:00

test_modeling_ibert.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_layoutlm.py

make style (#11442 )

2021-04-26 13:50:34 +02:00

test_modeling_led.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_longformer.py

Add multi-class, multi-label and regression to transformers (#11012 )

2021-05-04 02:23:40 -04:00

test_modeling_luke.py

Fix integration tests (#12066 )

2021-06-08 05:21:38 -04:00

test_modeling_lxmert.py

make style (#11442 )

2021-04-26 13:50:34 +02:00

test_modeling_m2m_100.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_marian.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_mbart.py

Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027 )

2021-06-08 13:58:38 +01:00

test_modeling_megatron_bert.py

A cleaner and more scalable implementation of symbolic tracing (#11763 )

2021-05-20 18:02:29 +02:00

test_modeling_megatron_gpt2.py

Fix megatron_gpt2 attention block's causal mask (#12007 )

2021-06-14 04:57:55 -04:00

test_modeling_mobilebert.py

A cleaner and more scalable implementation of symbolic tracing (#11763 )

2021-05-20 18:02:29 +02:00

test_modeling_mpnet.py

Add MP Net 2 (#9004 )

2020-12-09 10:32:43 -05:00

test_modeling_mt5.py

2020-12-07 18:36:34 -05:00

test_modeling_openai.py

2020-12-07 18:36:34 -05:00

test_modeling_pegasus.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_prophetnet.py

Fix usage of head masks by PT encoder-decoder models' generate() function (#11621 )

2021-05-19 00:44:53 +01:00

test_modeling_rag.py

[RAG] Fix rag from pretrained question encoder generator behavior (#11962 )

2021-06-02 09:17:14 +01:00

test_modeling_reformer.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_roberta.py

Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027 )

2021-06-08 13:58:38 +01:00

test_modeling_roformer.py

Add new model RoFormer (use rotary position embedding ) (#11684 )

2021-05-20 08:00:34 -04:00

test_modeling_speech_to_text.py

Fix cross-attention head mask for Torch encoder-decoder models (#10605 )

2021-04-23 18:58:06 +02:00

test_modeling_squeezebert.py

Add multi-class, multi-label and regression to transformers (#11012 )

2021-05-04 02:23:40 -04:00

test_modeling_t5.py

Flax T5 (#12150 )

2021-06-23 13:13:32 +01:00

test_modeling_tapas.py

Fix tapas issue (#12063 )

2021-06-08 05:22:31 -04:00

test_modeling_tf_albert.py

Add support for multiple models for one config in auto classes (#11150 )

2021-04-08 18:41:36 -04:00

test_modeling_tf_auto.py

Add support for multiple models for one config in auto classes (#11150 )

2021-04-08 18:41:36 -04:00

test_modeling_tf_bart.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_bert.py

Add support for multiple models for one config in auto classes (#11150 )

2021-04-08 18:41:36 -04:00

test_modeling_tf_blenderbot_small.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_blenderbot.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_bort.py

ADD BORT (#9813 )

2021-01-27 21:25:11 +03:00

test_modeling_tf_camembert.py

2020-12-07 18:36:34 -05:00

test_modeling_tf_common.py

Clean push to hub API (#12187 )

2021-06-23 10:11:19 -04:00

test_modeling_tf_convbert.py

ConvBERT fix torch <> tf weights conversion (#10314 )

2021-02-24 14:55:34 +03:00

test_modeling_tf_ctrl.py

Make TF CTRL compliant with XLA and AMP (#10209 )

2021-02-17 18:54:15 +01:00

test_modeling_tf_distilbert.py

Check TF ops for ONNX compliance (#10025 )

2021-02-15 07:55:10 -05:00

test_modeling_tf_dpr.py

Check TF ops for ONNX compliance (#10025 )

2021-02-15 07:55:10 -05:00

test_modeling_tf_electra.py

Check TF ops for ONNX compliance (#10025 )

2021-02-15 07:55:10 -05:00

test_modeling_tf_flaubert.py

Making TF XLM-like models XLA and AMP compliant (#10211 )

2021-02-17 18:02:48 +01:00

test_modeling_tf_funnel.py

Skip Funnel test

2021-05-05 12:38:01 +02:00

test_modeling_tf_gpt2.py

Making TF GPT2 compliant with XLA and AMP (#10230 )

2021-02-18 09:36:01 +01:00

test_modeling_tf_layoutlm.py

Layout lm tf 2 (#10636 )

2021-03-25 12:32:38 -04:00

test_modeling_tf_led.py

Fix usage of head masks by TF encoder-decoder models' generate() function (#11775 )

2021-05-26 14:02:44 +01:00

test_modeling_tf_longformer.py

Making TF Longformer-like models compliant with AMP (#10233 )

2021-02-22 15:41:56 +01:00

test_modeling_tf_lxmert.py

Making TF Lxmert model compliant with AMP (#10257 )

2021-02-19 06:54:14 -05:00

test_modeling_tf_marian.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_mbart.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_mobilebert.py

Making TF MobileBert model compliant with AMP (#10259 )

2021-02-19 06:55:25 -05:00

test_modeling_tf_mpnet.py

Making TF MPNet model compliant with XLA (#10260 )

2021-02-19 06:56:41 -05:00

test_modeling_tf_mt5.py

Adjust loss difference (#10669 )

2021-03-12 09:09:46 +03:00

test_modeling_tf_openai.py

Making TF OpenAI GPT model compliant with AMP and XLA (#10261 )

2021-02-19 09:33:25 -05:00

test_modeling_tf_pegasus.py

TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699 )

2021-04-26 14:16:21 +02:00

test_modeling_tf_pytorch.py

Optional layers (#8961 )

2020-12-08 09:14:09 -05:00

test_modeling_tf_rag.py

make rag tests smaller (#10679 )

2021-03-15 10:07:12 +03:00

test_modeling_tf_roberta.py

Check TF ops for ONNX compliance (#10025 )

2021-02-15 07:55:10 -05:00

test_modeling_tf_roformer.py

Add new model RoFormer (use rotary position embedding ) (#11684 )

2021-05-20 08:00:34 -04:00

test_modeling_tf_t5.py

ByT5 model (#11971 )

2021-06-01 19:07:37 +01:00

test_modeling_tf_transfo_xl.py

Making TF TransfoXL model compliant with AMP (#10264 )

2021-02-19 06:58:07 -05:00

test_modeling_tf_wav2vec2.py

Add out of vocabulary error to ASR models (#12288 )

2021-06-29 08:57:46 +01:00

test_modeling_tf_xlm_roberta.py

2020-12-07 18:36:34 -05:00

test_modeling_tf_xlm.py

Making TF XLM-like models XLA and AMP compliant (#10211 )

2021-02-17 18:02:48 +01:00

test_modeling_tf_xlnet.py

Check TF ops for ONNX compliance (#10025 )

2021-02-15 07:55:10 -05:00

test_modeling_transfo_xl.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_visual_bert.py

VisualBERT (#10534 )

2021-06-02 18:13:08 +05:30

test_modeling_vit.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_modeling_wav2vec2.py

Add out of vocabulary error to ASR models (#12288 )

2021-06-29 08:57:46 +01:00

test_modeling_xlm_prophetnet.py

Ci test tf super slow (#8007 )

2020-10-30 10:25:48 -04:00

test_modeling_xlm_roberta.py

2020-12-07 18:36:34 -05:00

test_modeling_xlm.py

Add multi-class, multi-label and regression to transformers (#11012 )

2021-05-04 02:23:40 -04:00

test_modeling_xlnet.py

Pytorch - Lazy initialization of models (#11471 )

2021-05-05 17:22:20 +02:00

test_offline.py

offline mode for firewalled envs (part 2) (#10569 )

2021-03-08 08:52:20 -08:00

test_onnx.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_optimization_tf.py

Use stable functions (#9369 )

2021-01-05 03:58:26 -05:00

test_optimization.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_pipelines_automatic_speech_recognition.py

Adding AutomaticSpeechRecognitionPipeline. (#11337 )

2021-04-30 11:54:08 +02:00

test_pipelines_common.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_conversational.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_pipelines_feature_extraction.py

2020-12-07 18:36:34 -05:00

test_pipelines_fill_mask.py

Temporarily revert the fill-mask improvements.

2021-06-23 17:46:24 +02:00

test_pipelines_image_classification.py

Pipeline update & tests (#12207 )

2021-06-17 09:41:16 +02:00

test_pipelines_question_answering.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_summarization.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_pipelines_table_question_answering.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_text2text_generation.py

2020-12-07 18:36:34 -05:00

test_pipelines_text_classification.py

Adding pipeline task aliases. (#11247 )

2021-04-15 09:51:24 +02:00

test_pipelines_text_generation.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_token_classification.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_translation.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_pipelines_zero_shot.py

Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752 )

2021-05-18 09:47:28 -04:00

test_processor_clip.py

CLIP (#11445 )

2021-05-12 13:48:15 +05:30

test_processor_speech_to_text.py

Auto feature extractor (#11097 )

2021-04-06 19:20:08 -04:00

test_processor_wav2vec2.py

[PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324 )

2021-02-25 17:42:46 +03:00

test_retrieval_rag.py

RAG-2nd2end-revamp (#11893 )

2021-06-01 07:32:26 +01:00

test_sequence_feature_extraction_common.py

Speech2TextTransformer (#10175 )

2021-03-10 21:42:04 +05:30

test_skip_decorators.py

[testing] rename skip targets + docs (#7863 )

2020-10-20 04:39:13 -04:00

test_tokenization_albert.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_auto.py

AutoTokenizer: infer the class from the tokenizer config if possible (#12208 )

2021-06-17 12:39:22 -04:00

test_tokenization_bart.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_barthez.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_bert_generation.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_bert_japanese.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_bert.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_bertweet.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_big_bird.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_blenderbot.py

[PyTorch Bart] Split Bart into different models (#9343 )

2021-01-05 22:00:05 +01:00

test_tokenization_byt5.py

ByT5 model (#11971 )

2021-06-01 19:07:37 +01:00

test_tokenization_camembert.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_clip.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_common.py

Easily train a new fast tokenizer from a given one (#12361 )

2021-06-29 15:00:08 -04:00

test_tokenization_cpm.py

Add a special tokenizer for CPM model (#11068 )

2021-04-10 02:07:47 +08:00

test_tokenization_ctrl.py

Refactor prepare_seq2seq_batch (#9524 )

2021-01-12 18:19:38 -05:00

test_tokenization_deberta_v2.py

fix deberta 2 tokenizer integration test (#12017 )

2021-06-07 04:55:55 -04:00

test_tokenization_deberta.py

Implement Fast Tokenization for Deberta (#11387 )

2021-04-30 08:08:15 -04:00

test_tokenization_distilbert.py

2020-12-07 18:36:34 -05:00

test_tokenization_dpr.py

[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659 )

2020-10-18 20:51:24 +02:00

test_tokenization_fast.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_fsmt.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_funnel.py

Reorganize repo (#8580 )

2020-11-16 21:43:42 -05:00

test_tokenization_gpt2.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_herbert.py

Reorganize repo (#8580 )

2020-11-16 21:43:42 -05:00

test_tokenization_layoutlm.py

make style (#11442 )

2021-04-26 13:50:34 +02:00

test_tokenization_luke.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_lxmert.py

2020-12-07 18:36:34 -05:00

test_tokenization_m2m_100.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_marian.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_mbart50.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_mbart.py

Enabling multilingual models for translation pipelines. (#10536 )

2021-04-16 11:31:35 +02:00

test_tokenization_mpnet.py

[MPNet] Add slow to fast tokenizer converter (#9233 )

2020-12-21 15:41:34 +01:00

test_tokenization_openai.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_pegasus.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_phobert.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_prophetnet.py

Deprecate prepare_seq2seq_batch (#10287 )

2021-02-22 12:36:16 -05:00

test_tokenization_rag.py

2020-12-07 18:36:34 -05:00

test_tokenization_reformer.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_roberta.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_roformer.py

Add new model RoFormer (use rotary position embedding ) (#11684 )

2021-05-20 08:00:34 -04:00

test_tokenization_small_blenderbot.py

Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810 )

2021-06-14 11:58:44 +02:00

test_tokenization_speech_to_text.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_squeezebert.py

[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659 )

2020-10-18 20:51:24 +02:00

test_tokenization_t5.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_tapas.py

Enforce string-formatting with f-strings (#10980 )

2021-03-31 10:00:27 -04:00

test_tokenization_transfo_xl.py

Refactor prepare_seq2seq_batch (#9524 )

2021-01-12 18:19:38 -05:00

test_tokenization_utils.py

Patch recursive import (#11812 )

2021-05-21 06:50:01 -04:00

test_tokenization_wav2vec2.py

Fix failing test on Windows Platform (#11589 )

2021-05-20 19:54:23 -04:00

test_tokenization_xlm_prophetnet.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_xlm_roberta.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_tokenization_xlm.py

make style (#11442 )

2021-04-26 13:50:34 +02:00

test_tokenization_xlnet.py

Add regression tests for slow sentencepiece tokenizers. (#11737 )

2021-06-01 09:24:39 -04:00

test_trainer_callback.py

fix: The 'warn' method is deprecated (#11105 )

2021-04-07 09:20:06 -04:00

test_trainer_distributed.py

[testing] ensure concurrent pytest workers use a unique port for torch.dist (#12166 )

2021-06-15 11:12:59 -07:00

test_trainer_seq2seq.py

fix bert2bert test (#10063 )

2021-02-08 16:04:28 +03:00

test_trainer_tpu.py

Examples reorg (#11350 )

2021-04-21 11:11:20 -04:00

test_trainer_utils.py

[style] consistent nn. and nn.functional: part 3 tests (#12155 )

2021-06-14 12:18:22 -07:00

test_trainer.py

Clean push to hub API (#12187 )

2021-06-23 10:11:19 -04:00

test_utils_check_copies.py

Fix failing test on Windows Platform (#11589 )

2021-05-20 19:54:23 -04:00

test_versions_utils.py

rm require_version_examples (#12088 )

2021-06-09 11:02:52 -07:00