SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
135791e8ef12802ceb21a4abbb3a93f7da1bf390
HuggingFace_transformer/tests
History
Funtowicz Morgan 135791e8ef Add pad_to_multiple_of on tokenizers (reimport) (#5054) 
* Add new parameter `pad_to_multiple_of` on tokenizers.

* unittest for pad_to_multiple_of

* Add .name when logging enum.

* Fix missing .items() on dict in tests.

* Add special check + warning if the tokenizer doesn't have proper pad_token.

* Use the correct logger format specifier.

* Ensure tokenizer with no pad_token do not modify the underlying padding strategy.

* Skip test if tokenizer doesn't have pad_token

* Fix RobertaTokenizer on empty input

* Format.

Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>

* fix and updating to simpler API

Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
2020-06-26 11:55:57 +02:00
..
fixtures
BIG Reorganize examples (#4213)
2020-05-07 13:48:44 -04:00
__init__.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
test_activations.py
Reformer (#3351)
2020-05-07 10:17:01 +02:00
test_benchmark_tf.py
[Benchmark] Extend Benchmark to all model type extensions (#5241)
2020-06-24 15:11:42 +02:00
test_benchmark.py
[Benchmark] Extend Benchmark to all model type extensions (#5241)
2020-06-24 15:11:42 +02:00
test_configuration_auto.py
Map configs to models and tokenizers
2020-01-13 23:11:44 +00:00
test_configuration_common.py
Pass kwargs to configuration (#3147)
2020-03-05 17:16:57 -05:00
test_doc_samples.py
Refactor Code samples; Test code samples (#5036)
2020-06-25 16:46:00 -04:00
test_hf_api.py
[cli] {login, upload, s3} display more helpful error messages
2020-04-30 12:51:06 -04:00
test_hf_argparser.py
[Benchmark] Memory benchmark utils (#4198)
2020-05-27 23:22:16 +02:00
test_model_card.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
test_modeling_albert.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_auto.py
[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
2020-06-25 17:24:28 +02:00
test_modeling_bart.py
[Use cache] Align logic of use_cache with output_attentions and output_hidden_states (#5194)
2020-06-24 16:09:17 +02:00
test_modeling_bert.py
[AutoModel] Split AutoModelWithLMHead into clm, mlm, encoder-decoder (#4933)
2020-06-12 10:01:49 +02:00
test_modeling_camembert.py
Fix slow gpu tests lysandre (#4487)
2020-05-20 11:59:45 -04:00
test_modeling_common.py
[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
2020-06-25 17:24:28 +02:00
test_modeling_ctrl.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_distilbert.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_electra.py
ElectraForMultipleChoice (#4954)
2020-06-18 14:59:35 -04:00
test_modeling_encoder_decoder.py
Split LMBert model in two (#4874)
2020-06-10 18:26:42 -04:00
test_modeling_flaubert.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_gpt2.py
[Use cache] Align logic of use_cache with output_attentions and output_hidden_states (#5194)
2020-06-24 16:09:17 +02:00
test_modeling_longformer.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_marian.py
[marian tests ] pass device to pipeline (#4815)
2020-06-06 00:52:17 -04:00
test_modeling_mobilebert.py
[fix] remove unused import (#5206)
2020-06-22 23:39:04 -04:00
test_modeling_openai.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_reformer.py
[Reformer] Improved memory if input is shorter than chunk length (#4720)
2020-06-02 23:08:39 +02:00
test_modeling_roberta.py
[bart] add config.extra_pos_embeddings to facilitate reuse (#5190)
2020-06-23 11:35:42 -04:00
test_modeling_t5.py
[Use cache] Align logic of use_cache with output_attentions and output_hidden_states (#5194)
2020-06-24 16:09:17 +02:00
test_modeling_tf_albert.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_tf_auto.py
[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
2020-06-25 17:24:28 +02:00
test_modeling_tf_bert.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_tf_camembert.py
Add camembert integration tests (#3375)
2020-03-24 10:18:37 +01:00
test_modeling_tf_common.py
Cleaning TensorFlow models (#5229)
2020-06-24 11:37:20 -04:00
test_modeling_tf_ctrl.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_tf_distilbert.py
Cleaning TensorFlow models (#5229)
2020-06-24 11:37:20 -04:00
test_modeling_tf_electra.py
Cleaning TensorFlow models (#5229)
2020-06-24 11:37:20 -04:00
test_modeling_tf_flaubert.py
Tensorflow improvements (#4530)
2020-06-04 19:45:53 -04:00
test_modeling_tf_gpt2.py
[Use cache] Align logic of use_cache with output_attentions and output_hidden_states (#5194)
2020-06-24 16:09:17 +02:00
test_modeling_tf_mobilebert.py
Add MobileBert (#4901)
2020-06-19 16:38:36 -04:00
test_modeling_tf_openai_gpt.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_tf_roberta.py
Cleaning TensorFlow models (#5229)
2020-06-24 11:37:20 -04:00
test_modeling_tf_t5.py
[Use cache] Align logic of use_cache with output_attentions and output_hidden_states (#5194)
2020-06-24 16:09:17 +02:00
test_modeling_tf_transfo_xl.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_tf_xlm_roberta.py
Tensorflow improvements (#4530)
2020-06-04 19:45:53 -04:00
test_modeling_tf_xlm.py
Refactor Code samples; Test code samples (#5036)
2020-06-25 16:46:00 -04:00
test_modeling_tf_xlnet.py
Cleaning TensorFlow models (#5229)
2020-06-24 11:37:20 -04:00
test_modeling_transfo_xl.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_xlm_roberta.py
[ci] Re-run integration ground truth from fairseq
2020-03-03 15:31:40 -05:00
test_modeling_xlm.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_modeling_xlnet.py
[cleanup] Hoist ModelTester objects to top level (#4939)
2020-06-16 08:03:43 -04:00
test_onnx.py
Fix onnx export input names order (#4641)
2020-06-01 16:12:48 +02:00
test_optimization_tf.py
Tensorflow improvements (#4530)
2020-06-04 19:45:53 -04:00
test_optimization.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
test_pipelines.py
[test_pipelines] Mark tests > 10s @slow, small speedups (#4421)
2020-05-18 12:23:21 -04:00
test_tokenization_albert.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_auto.py
[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
2020-06-25 17:24:28 +02:00
test_tokenization_bert_japanese.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_bert.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_common.py
Add pad_to_multiple_of on tokenizers (reimport) (#5054)
2020-06-26 11:55:57 +02:00
test_tokenization_ctrl.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_distilbert.py
[cleanup] test_tokenization_common.py (#4390)
2020-05-19 10:46:55 -04:00
test_tokenization_fast.py
[tokenizers] Several small improvements and bug fixes (#5287)
2020-06-25 22:17:14 +02:00
test_tokenization_gpt2.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_marian.py
Fix marian tokenizer save pretrained (#5043)
2020-06-16 09:48:19 -04:00
test_tokenization_openai.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_roberta.py
[tokenizers] Several small improvements and bug fixes (#5287)
2020-06-25 22:17:14 +02:00
test_tokenization_t5.py
[cleanup] test_tokenization_common.py (#4390)
2020-05-19 10:46:55 -04:00
test_tokenization_transfo_xl.py
Added feature to move added tokens in vocabulary for Transformer-XL (#4953)
2020-06-22 15:40:52 +02:00
test_tokenization_utils.py
Ability to pickle/unpickle BatchEncoding pickle (reimport) (#5039)
2020-06-16 09:25:25 +02:00
test_tokenization_xlm_roberta.py
[cleanup] test_tokenization_common.py (#4390)
2020-05-19 10:46:55 -04:00
test_tokenization_xlm.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_xlnet.py
[cleanup] test_tokenization_common.py (#4390)
2020-05-19 10:46:55 -04:00
test_trainer_distributed.py
[Tokenization] Fix #5181 - make #5155 more explicit - move back the default logging level in tests to WARNING (#5252)
2020-06-25 17:24:28 +02:00
test_trainer.py
Fix #5114 (#5122)
2020-06-18 19:20:04 -04:00
utils.py
Fix nn.DataParallel compatibility in PyTorch 1.5 (#4300)
2020-05-18 20:34:50 -04:00