SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
cb276b41deaa97088e6d56fdca9ceb17db16ca19
HuggingFace_transformer/tests
History
RafaelWO cb276b41de Transformer-XL: Improved tokenization with sacremoses (#6322) 
* Improved tokenization with sacremoses

 * The TransfoXLTokenizer is now using sacremoses for tokenization
 * Added tokenization of comma-separated and floating point numbers.
 * Removed prepare_for_tokenization() from tokenization_transfo_xl.py because punctuation is handled by sacremoses
 * Added corresponding tests
 * Removed test comapring TransfoXLTokenizer and TransfoXLTokenizerFast
 * Added deprecation warning to TransfoXLTokenizerFast

* isort change

Co-authored-by: Teven <teven.lescao@gmail.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
2020-08-28 09:56:17 -04:00
..
fixtures
BIG Reorganize examples (#4213)
2020-05-07 13:48:44 -04:00
__init__.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
conftest.py
enable easy checkout switch (#5645)
2020-07-31 04:34:46 -04:00
test_activations.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_benchmark_tf.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_benchmark.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_cli.py
[transformers-cli] fix logger getter (#6777)
2020-08-27 20:01:17 -04:00
test_configuration_auto.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_configuration_common.py
Pass kwargs to configuration (#3147)
2020-03-05 17:16:57 -05:00
test_data_collator.py
Add tests to Trainer (#6605)
2020-08-20 11:13:50 -04:00
test_doc_samples.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_hf_api.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_hf_argparser.py
parse arguments from dict (#4869)
2020-07-31 04:44:23 -04:00
test_logging.py
Centralize logging (#6434)
2020-08-26 11:10:36 -04:00
test_model_card.py
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
test_modeling_albert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_auto.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_bart.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_bert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_camembert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_common.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_ctrl.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_distilbert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_dpr.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_electra.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_encoder_decoder.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_flaubert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_gpt2.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_longformer.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_marian.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_mbart.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_mobilebert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_openai.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_pegasus.py
Fix pegasus-xsum integration test (#6726)
2020-08-25 14:06:28 -04:00
test_modeling_reformer.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_roberta.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_t5.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_albert.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_tf_auto.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_tf_bert.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_tf_camembert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_common.py
[model_cards] Fix tiny typos
2020-08-26 23:16:06 +02:00
test_modeling_tf_ctrl.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_distilbert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_electra.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_flaubert.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_gpt2.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_longformer.py
[TF Longformer] Improve Speed for TF Longformer (#6447)
2020-08-26 14:55:41 -04:00
test_modeling_tf_mobilebert.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_tf_openai.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_roberta.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_t5.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_transfo_xl.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_xlm_roberta.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_tf_xlm.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_tf_xlnet.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_transfo_xl.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_xlm_roberta.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_modeling_xlm.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_modeling_xlnet.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_onnx.py
Fix flaky ONNX tests (#6531)
2020-08-17 09:04:35 -04:00
test_optimization_tf.py
Update repo to isort v5 (#6686)
2020-08-24 11:03:01 -04:00
test_optimization.py
Format
2020-08-27 18:31:51 +02:00
test_pipelines.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_tokenization_albert.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_auto.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_tokenization_bert_japanese.py
Support additional dictionaries for BERT Japanese tokenizers (#6515)
2020-08-17 12:00:23 +08:00
test_tokenization_bert.py
Add strip_accents to basic BertTokenizer. (#6280)
2020-08-06 18:52:28 +08:00
test_tokenization_common.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_tokenization_ctrl.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_distilbert.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_tokenization_dpr.py
Fix tests imports dpr (#5576)
2020-07-07 16:35:12 +02:00
test_tokenization_fast.py
Transformer-XL: Improved tokenization with sacremoses (#6322)
2020-08-28 09:56:17 -04:00
test_tokenization_gpt2.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_marian.py
rename prepare_translation_batch -> prepare_seq2seq_batch (#6103)
2020-08-11 15:57:07 -04:00
test_tokenization_mbart.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_tokenization_openai.py
[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)
2020-06-15 17:12:51 -04:00
test_tokenization_pegasus.py
PegasusForConditionalGeneration (torch version) (#6340)
2020-08-11 14:31:23 -04:00
test_tokenization_reformer.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_tokenization_roberta.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_tokenization_t5.py
Black 20 release
2020-08-26 17:20:22 +02:00
test_tokenization_transfo_xl.py
Transformer-XL: Improved tokenization with sacremoses (#6322)
2020-08-28 09:56:17 -04:00
test_tokenization_utils.py
Fixes to make life easier with the nlp library (#6423)
2020-08-12 08:00:56 -04:00
test_tokenization_xlm_roberta.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_tokenization_xlm.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_tokenization_xlnet.py
Move tests/utils.py -> transformers/testing_utils.py (#5350)
2020-07-01 10:31:17 -04:00
test_trainer_distributed.py
Add tests to Trainer (#6605)
2020-08-20 11:13:50 -04:00
test_trainer.py
[testing] replace hardcoded paths to allow running tests from anywhere (#6523)
2020-08-27 12:22:18 -04:00