Deniz 32e94cff64 tf add resize_token_embeddings method (#4351) ...

* resize token embeddings

* add tokens

* add tokens

* add tokens

* add t5 token method

* add t5 token method

* add t5 token method

* typo

* debugging input

* debugging input

* debug

* debug

* debug

* trying to set embedding tokens properly

* set embeddings for generation head too

* set embeddings for generation head too

* debugging

* debugging

* enable generation

* add base method

* add base method

* add base method

* return logits in the main call

* reverting to generation

* revert back

* set embeddings for the bert main layer

* description

* fix conflicts

* logging

* set base model as self

* refactor

* tf_bert add method

* tf_bert add method

* tf_bert add method

* tf_bert add method

* tf_bert add method

* tf_bert add method

* tf_bert add method

* tf_bert add method

* v0

* v0

* finalize

* final

* black

* add tests

* revert back the emb call

* comments

* comments

* add the second test

* add vocab size condig

* add tf models

* add tf models. add common tests

* remove model specific embedding tests

* stylish

* remove files

* stylez

* Update src/transformers/modeling_tf_transfo_xl.py

change the error.

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* adding unchanged weight test

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

2020-06-18 18:41:26 -04:00

..

fixtures

BIG Reorganize examples (#4213)

2020-05-07 13:48:44 -04:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_activations.py

Reformer (#3351)

2020-05-07 10:17:01 +02:00

test_benchmark.py

[Benchmark] add tpu and torchscipt for benchmark (#4850)

2020-06-09 23:12:43 +02:00

test_configuration_auto.py

Map configs to models and tokenizers

2020-01-13 23:11:44 +00:00

test_configuration_common.py

Pass kwargs to configuration (#3147)

2020-03-05 17:16:57 -05:00

test_doc_samples.py

make style

2020-03-04 20:18:07 -05:00

test_hf_api.py

[cli] {login, upload, s3} display more helpful error messages

2020-04-30 12:51:06 -04:00

test_hf_argparser.py

[Benchmark] Memory benchmark utils (#4198)

2020-05-27 23:22:16 +02:00

test_model_card.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_modeling_albert.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_auto.py

[AutoModel] Split AutoModelWithLMHead into clm, mlm, encoder-decoder (#4933)

2020-06-12 10:01:49 +02:00

test_modeling_bart.py

[Bart] Question Answering Model is added to tests (#5024)

2020-06-15 22:50:09 +02:00

test_modeling_bert.py

[AutoModel] Split AutoModelWithLMHead into clm, mlm, encoder-decoder (#4933)

2020-06-12 10:01:49 +02:00

test_modeling_camembert.py

Fix slow gpu tests lysandre (#4487)

2020-05-20 11:59:45 -04:00

test_modeling_common.py

[Bart] Question Answering Model is added to tests (#5024)

2020-06-15 22:50:09 +02:00

test_modeling_ctrl.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_distilbert.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_electra.py

ElectraForMultipleChoice (#4954)

2020-06-18 14:59:35 -04:00

test_modeling_encoder_decoder.py

Split LMBert model in two (#4874)

2020-06-10 18:26:42 -04:00

test_modeling_flaubert.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_gpt2.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_longformer.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_marian.py

[marian tests ] pass device to pipeline (#4815)

2020-06-06 00:52:17 -04:00

test_modeling_openai.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_reformer.py

[Reformer] Improved memory if input is shorter than chunk length (#4720)

2020-06-02 23:08:39 +02:00

test_modeling_roberta.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_t5.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_albert.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_auto.py

Kill model archive maps (#4636)

2020-06-02 09:39:33 -04:00

test_modeling_tf_bert.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_camembert.py

Add camembert integration tests (#3375)

2020-03-24 10:18:37 +01:00

test_modeling_tf_common.py

tf add resize_token_embeddings method (#4351)

2020-06-18 18:41:26 -04:00

test_modeling_tf_ctrl.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_distilbert.py

tf add resize_token_embeddings method (#4351)

2020-06-18 18:41:26 -04:00

test_modeling_tf_electra.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_flaubert.py

Tensorflow improvements (#4530)

2020-06-04 19:45:53 -04:00

test_modeling_tf_gpt2.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_openai_gpt.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_roberta.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_t5.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_transfo_xl.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_xlm_roberta.py

Tensorflow improvements (#4530)

2020-06-04 19:45:53 -04:00

test_modeling_tf_xlm.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_tf_xlnet.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_transfo_xl.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_xlm_roberta.py

[ci] Re-run integration ground truth from fairseq

2020-03-03 15:31:40 -05:00

test_modeling_xlm.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_modeling_xlnet.py

[cleanup] Hoist ModelTester objects to top level (#4939)

2020-06-16 08:03:43 -04:00

test_onnx.py

Fix onnx export input names order (#4641)

2020-06-01 16:12:48 +02:00

test_optimization_tf.py

Tensorflow improvements (#4530)

2020-06-04 19:45:53 -04:00

test_optimization.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_pipelines.py

[test_pipelines] Mark tests > 10s @slow, small speedups (#4421)

2020-05-18 12:23:21 -04:00

test_tokenization_albert.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_auto.py

Test correct tokenizers after default switch (#3003)

2020-02-24 18:45:53 -05:00

test_tokenization_bert_japanese.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_bert.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_common.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_ctrl.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_distilbert.py

[cleanup] test_tokenization_common.py (#4390)

2020-05-19 10:46:55 -04:00

test_tokenization_fast.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_gpt2.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_marian.py

Fix marian tokenizer save pretrained (#5043)

2020-06-16 09:48:19 -04:00

test_tokenization_openai.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_roberta.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_t5.py

[cleanup] test_tokenization_common.py (#4390)

2020-05-19 10:46:55 -04:00

test_tokenization_transfo_xl.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_utils.py

Ability to pickle/unpickle BatchEncoding pickle (reimport) (#5039)

2020-06-16 09:25:25 +02:00

test_tokenization_xlm_roberta.py

[cleanup] test_tokenization_common.py (#4390)

2020-05-19 10:46:55 -04:00

test_tokenization_xlm.py

[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)

2020-06-15 17:12:51 -04:00

test_tokenization_xlnet.py

[cleanup] test_tokenization_common.py (#4390)

2020-05-19 10:46:55 -04:00

test_trainer_distributed.py

Make DataCollator a callable (#5015)

2020-06-15 11:58:33 -04:00

test_trainer.py

Make default_data_collator more flexible and deprecate old behavior (#5060)

2020-06-17 15:24:51 -04:00

utils.py

Fix nn.DataParallel compatibility in PyTorch 1.5 (#4300)

2020-05-18 20:34:50 -04:00