Thomas Wolf 827d6d6ef0 Cleanup fast tokenizers integration (#3706) ...

* First pass on utility classes and python tokenizers

* finishing cleanup pass

* style and quality

* Fix tests

* Updating following @mfuntowicz comment

* style and quality

* Fix Roberta

* fix batch_size/seq_length inBatchEncoding

* add alignement methods + tests

* Fix OpenAI and Transfo-XL tokenizers

* adding trim_offsets=True default for GPT2 et RoBERTa

* style and quality

* fix tests

* add_prefix_space in roberta

* bump up tokenizers to rc7

* style

* unfortunately tensorfow does like these - removing shape/seq_len for now

* Update src/transformers/tokenization_utils.py

Co-Authored-By: Stefan Schweter <stefan@schweter.it>

* Adding doc and docstrings

* making flake8 happy

Co-authored-by: Stefan Schweter <stefan@schweter.it>

2020-04-18 13:43:57 +02:00

..

fixtures

AutoConfig + other Auto classes honor model_type

2020-01-11 02:46:17 +00:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_activations.py

get_activation('relu') provides a simple mapping from strings i… (#2807)

2020-02-13 08:28:33 -05:00

test_configuration_auto.py

Map configs to models and tokenizers

2020-01-13 23:11:44 +00:00

test_configuration_common.py

Pass kwargs to configuration (#3147)

2020-03-05 17:16:57 -05:00

test_doc_samples.py

make style

2020-03-04 20:18:07 -05:00

test_hf_api.py

[model upload] Support for organizations

2020-03-09 17:33:57 -04:00

test_hf_argparser.py

[examples] Generate argparsers from type hints on dataclasses (#3669)

2020-04-10 12:21:58 -04:00

test_model_card.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_modeling_albert.py

Added test for AlbertForTokenClassification

2020-02-27 12:24:20 +01:00

test_modeling_auto.py

Add missing token classification for XLM (#3277)

2020-03-26 10:22:13 -04:00

test_modeling_bart.py

Multilingual BART - (#3602)

2020-04-10 11:25:39 -04:00

test_modeling_bert.py

BERT decoder: Fix causal mask dtype.

2020-02-11 15:19:22 -05:00

test_modeling_camembert.py

Add camembert integration tests (#3375)

2020-03-24 10:18:37 +01:00

test_modeling_common.py

[Config, Caching] Remove output_past everywhere and replace by use_cache argument (#3734)

2020-04-14 14:40:28 -04:00

test_modeling_ctrl.py

test ctrl

2020-03-09 13:58:01 +00:00

test_modeling_distilbert.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_modeling_electra.py

ELECTRA (#3257)

2020-04-03 14:10:54 -04:00

test_modeling_flaubert.py

Correct slow test

2020-02-04 18:05:35 -05:00

test_modeling_gpt2.py

Clean special token init in modeling_....py (#3264)

2020-03-20 21:41:04 +01:00

test_modeling_openai.py

updated all tests

2020-03-08 15:29:10 +01:00

test_modeling_roberta.py

[ci] Re-run integration ground truth from fairseq

2020-03-03 15:31:40 -05:00

test_modeling_t5.py

Higher tolerance for past testing in T5 (#3843)

2020-04-17 11:25:14 -04:00

test_modeling_tf_albert.py

Question Answering support for Albert and Roberta in TF (#3812)

2020-04-17 10:45:30 -04:00

test_modeling_tf_auto.py

Add AutoModelForPreTraining

2020-01-27 14:27:07 -05:00

test_modeling_tf_bert.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_modeling_tf_camembert.py

Add camembert integration tests (#3375)

2020-03-24 10:18:37 +01:00

test_modeling_tf_common.py

[Config, Caching] Remove output_past everywhere and replace by use_cache argument (#3734)

2020-04-14 14:40:28 -04:00

test_modeling_tf_ctrl.py

test ctrl

2020-03-09 13:58:01 +00:00

test_modeling_tf_distilbert.py

Definitive HeisenDistilBug fix

2020-01-27 12:09:58 -05:00

test_modeling_tf_electra.py

ELECTRA (#3257)

2020-04-03 14:10:54 -04:00

test_modeling_tf_gpt2.py

[TFT5, Cache] Add cache to TFT5 (#3772)

2020-04-16 16:14:52 +02:00

test_modeling_tf_openai_gpt.py

fixed all tests, still need to check ctrl tf and pt and xlm tf

2020-03-08 21:45:55 +01:00

test_modeling_tf_roberta.py

Question Answering support for Albert and Roberta in TF (#3812)

2020-04-17 10:45:30 -04:00

test_modeling_tf_t5.py

Higher tolerance for past testing in TF T5 (#3844)

2020-04-17 11:26:16 -04:00

test_modeling_tf_transfo_xl.py

Clean special token init in modeling_....py (#3264)

2020-03-20 21:41:04 +01:00

test_modeling_tf_xlm.py

fix typo in test xlm tf

2020-03-09 11:34:31 +01:00

test_modeling_tf_xlnet.py

fixed all tests, still need to check ctrl tf and pt and xlm tf

2020-03-08 21:45:55 +01:00

test_modeling_transfo_xl.py

Shift labels internally within TransfoXLLMHeadModel when called with labels (#3716)

2020-04-13 18:11:23 +02:00

test_modeling_xlm_roberta.py

[ci] Re-run integration ground truth from fairseq

2020-03-03 15:31:40 -05:00

test_modeling_xlm.py

Add missing token classification for XLM (#3277)

2020-03-26 10:22:13 -04:00

test_modeling_xlnet.py

fix xlnet & transfotests

2020-03-08 16:25:03 +01:00

test_optimization_tf.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_optimization.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_pipelines.py

clean pipelines (#3795)

2020-04-16 10:21:34 -04:00

test_tokenization_albert.py

💄 super

2020-01-15 18:33:50 -05:00

test_tokenization_auto.py

Test correct tokenizers after default switch (#3003)

2020-02-24 18:45:53 -05:00

test_tokenization_bert_japanese.py

BertJapaneseTokenizer accept options for mecab (#3566)

2020-04-03 11:12:19 -04:00

test_tokenization_bert.py

Tokenizers v3.0.0 (#3185)

2020-04-07 00:29:15 +02:00

test_tokenization_common.py

Update quotes

2020-04-09 09:09:00 -04:00

test_tokenization_ctrl.py

💄 super

2020-01-15 18:33:50 -05:00

test_tokenization_distilbert.py

Updating the TensorFlow models to work as expected with tokenizers v3.0.0 (#3684)

2020-04-08 16:22:44 -04:00

test_tokenization_fast.py

Cleanup fast tokenizers integration (#3706)

2020-04-18 13:43:57 +02:00

test_tokenization_gpt2.py

Tokenizers v3.0.0 (#3185)

2020-04-07 00:29:15 +02:00

test_tokenization_openai.py

💄 super

2020-01-15 18:33:50 -05:00

test_tokenization_roberta.py

Preserve spaces in GPT-2 tokenizers (#2778)

2020-02-13 13:29:43 -05:00

test_tokenization_t5.py

💄 super

2020-01-15 18:33:50 -05:00

test_tokenization_transfo_xl.py

💄 super

2020-01-15 18:33:50 -05:00

test_tokenization_utils.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_tokenization_xlm_roberta.py

XLM-R Tokenizer now passes common tests + Integration tests (#3198)

2020-03-18 09:52:49 -04:00

test_tokenization_xlm.py

XLM tokenizer should encode with bos token (#3791)

2020-04-17 11:28:55 -04:00

test_tokenization_xlnet.py

💄 super

2020-01-15 18:33:50 -05:00

utils.py

TF GPU CI (#3085)

2020-03-02 15:45:25 -05:00