SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
6,651 Commits 5 Branches 239 Tags
11655fafdd42eb56ad94e09ecd84d4dc2d1041ae
Commit Graph

1817 Commits

Author SHA1 Message Date
Patrick von Platen
11655fafdd remove feature extraction config (#10457) 2021-03-01 12:30:12 +03:00
Patrick von Platen
0234de8418 Add Fine-Tuning for Wav2Vec2 (#10145) 
* add encode labels function to tokenizer

* start adding finetuning

* init dropout

* upload

* correct convert script

* apply changes

* fix second typo

* make first dummy training run

* adapt convert script

* push confg for comparison

* remove conf

* finish training

* adapt data collator

* add research folder

* update according to fairseq feedback

* some minor corrections

* refactor masking indices a bit

* some minor changes

* clean tokenizer

* finish clean-up

* remove previous logic

* update run script

* correct training

* finish changes

* finish model

* correct bug

* fix training a bit more

* add some tests

* finish gradient checkpointing

* finish example

* correct gradient checkpointing

* improve tokenization method

* revert changes in tokenizer

* revert general change

* adapt fine-tuning

* update

* save intermediate test

* Update README.md

* finish finetuning

* delete conversion script

* Update src/transformers/models/wav2vec2/configuration_wav2vec2.py

* Update src/transformers/models/wav2vec2/processing_wav2vec2.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* finish wav2vec2 script

* finish wav2vec2 fine-tuning

* finalize test

* correct test

* adapt tests

* finish

* remove test file

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-03-01 12:13:17 +03:00
Tanmay Garg
256482ac92 Introduce save_strategy training argument (#10286) 
* Introduce save_strategy training argument

* deprecate EvaluationStrategy

* collapse EvaluationStrategy and LoggingStrategy into a single
  IntervalStrategy enum

* modify tests to use modified enum
 2021-02-27 19:34:22 -05:00
Stas Bekman
ee04b69822 [examples] better model example (#10427) 
* refactors

* typo
 2021-02-26 17:01:01 -08:00
Amog Kamsetty
a85eb616f7 Ray Tune Integration Bug Fixes (#10406) 
* fixes

* update resources

* formatting

* remove import

* add log statement

* use fstring

* add period

* Update src/transformers/integrations.py
 2021-02-26 19:06:08 -05:00
Patrick von Platen
d03695f3a2 [LED] Correct Docs (#10419) 
* correct docs

* correct tf model docs as well
 2021-02-26 17:53:28 +03:00
Mansi Mane
7fc686efb1 Sagemaker Model Parallel tensoboard writing fix (#10403) 
* Added tb fix

* Removed local rank condition

* Updated reference to args
 2021-02-26 08:04:55 -05:00
Sylvain Gugger
17b6e0d474 Fix run_glue evaluation when model has a label correspondence (#10401) 2021-02-25 15:30:38 -05:00
Sylvain Gugger
9d14be5c20 Add support for ZeRO-2/3 and ZeRO-offload in fairscale (#10354) 
* Ass support for ZeRO-2/3 and ZeRO-offload in fairscale

* Quality

* Rework from review comments

* Add doc

* Apply suggestions from code review

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

* Address review comments

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
 2021-02-25 11:07:53 -05:00
Lysandre Debut
88cc26dcd1 Ignore unexpected weights from PT conversion (#10397) 2021-02-25 10:42:27 -05:00
Sehoon Kim
63645b3b11 I-BERT model support (#10153) 
* IBertConfig, IBertTokentizer added

* IBert Model names moified

* tokenizer bugfix

* embedding -> QuantEmbedding

* quant utils added

* quant_mode added to configuration

* QuantAct added, Embedding layer + QuantAct addition

* QuantAct added

* unused path removed, QKV quantized

* self attention layer all quantized, except softmax

* temporarl commit

* all liner layers quantized

* quant_utils bugfix

* bugfix: requantization missing

* IntGELU added

* IntSoftmax added

* LayerNorm implemented

* LayerNorm implemented all

* names changed: roberta->ibert

* config not inherit from ROberta

* No support for CausalLM

* static quantization added, quantize_model.py removed

* import modules uncommented

* copyrights fixed

* minor bugfix

* quant_modules, quant_utils merged as one file

* import * fixed

* unused runfile removed

* make style run

* configutration.py docstring fixed

* refactoring: comments removed, function name fixed

* unused dependency removed

* typo fixed

* comments(Copied from), assertion string added

* refactoring: super(..) -> super(), etc.

* refactoring

* refarctoring

* make style

* refactoring

* cuda -> to(x.device)

* weight initialization removed

* QuantLinear set_param removed

* QuantEmbedding set_param removed

* IntLayerNorm set_param removed

* assert string added

* assertion error message fixed

* is_decoder removed

* enc-dec arguments/functions removed

* Converter removed

* quant_modules docstring fixed

* conver_slow_tokenizer rolled back

* quant_utils docstring fixed

* unused aruments e.g. use_cache removed from config

* weight initialization condition fixed

* x_min, x_max initialized with small values to avoid div-zero exceptions

* testing code for ibert

* test emb, linear, gelu, softmax added

* test ln and act added

* style reformatted

* force_dequant added

* error tests overrided

* make style

* Style + Docs

* force dequant tests added

* Fix fast tokenizer in init

* Fix doc

* Remove space

* docstring, IBertConfig, chunk_size

* test_modeling_ibert refactoring

* quant_modules.py refactoring

* e2e integration test added

* tokenizers removed

* IBertConfig added to tokenizer_auto.py

* bugfix

* fix docs & test

* fix style num 2

* final fixes

Co-authored-by: Sehoon Kim <sehoonkim@berkeley.edu>
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2021-02-25 10:06:42 -05:00
Patrick von Platen
cb38ffcc5e [PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324) 
* push to show

* small improvement

* small improvement

* Update src/transformers/feature_extraction_utils.py

* Update src/transformers/feature_extraction_utils.py

* implement base

* add common tests

* make all tests pass for wav2vec2

* make padding work & add more tests

* finalize feature extractor utils

* add call method to feature extraction

* finalize feature processor

* finish tokenizer

* finish general processor design

* finish tests

* typo

* remove bogus file

* finish docstring

* add docs

* finish docs

* small fix

* correct docs

* save intermediate

* load changes

* apply changes

* apply changes to doc

* change tests

* apply surajs recommend

* final changes

* Apply suggestions from code review

* fix typo

* fix import

* correct docstring
 2021-02-25 17:42:46 +03:00
mingruimingrui
894db6701e Bugfix: Removal of padding_idx in BartLearnedPositionalEmbedding (#10200) 
* Assumption of padding_idx <2 might not stand

* Use offset instead of 2

* Fix with black

* Change behavior to warning instead for backward compatibility.

* Fix with black

* Remove warning

* Make padding_idx non-required

* padding_idx fix for blenderbot

* padding_idx fix for blenderbot_small

* padding_idx fix for led

* padding_idx fix for mbart

* Remove extra whitespaces

* padding_idx fix for template

* Fix padding_idx passed to nn.Embedding mistake

* Fixed padding_idx passed to positional embedding in template

* Remove padding_idx from pytorch learned positional embeddings

* Remove accidentally added quotes

* Remove padding_idx from tf learned positional embeddings

* Remove zeroing of weights in __init__

Co-authored-by: Wang Ming Rui <mingrui.wang@C02CJTUYMD6M.local>
 2021-02-25 14:33:13 +03:00
Stas Bekman
bdbb2c756b [trainer] move secondary methods into a separate file (#10363) 
* move secondary methods into a separate file

* cleanup

* style
 2021-02-24 08:32:52 -08:00
Poedator
5f2a3d721c fix deprecated ref to tokenizer.max_len (#10220) 
This is to fix deprecated reference to `tokenizer.max_len` with `tokenizer.model_max_length` - similar to [issue 8739](https://github.com/huggingface/transformers/issues/8739) and [PR 8604](https://github.com/huggingface/transformers/pull/8604). 
Example [here](https://colab.research.google.com/gist/poedator/f8776349e5c625ce287fc6fcd312fa1e/tokenizer-max_len-error-in-transformers_glue.ipynb). The error happens when `glue_convert_examples_to_features` is called without `max_length` parameter specified. In that case line 119 with wrong reference gets called. This simple fix should  do it.
 2021-02-24 09:01:28 -05:00
Julien Plu
cdcdd5f03a Rework casts (#10274) 2021-02-24 08:38:29 -05:00
abhishek thakur
2d458b2c7d ConvBERT fix torch <> tf weights conversion (#10314) 
* convbert conversion test

* fin

* fin

* fin

* clean up tf<->pt conversion

* remove from_pt

Co-authored-by: patrickvonplaten <patrick.v.platen@gmail.com>
 2021-02-24 14:55:34 +03:00
Stas Bekman
3437d12134 [Trainer/Deepspeed] handle get_last_lr() before first step() (#10362) 
* handle get_last_lr() before first step()

* abstract away the lr getting logic

* cleanup

* add test

* move to utils
 2021-02-23 17:42:25 -08:00
Julien Chaumond
4a1ab7cb6c [bert-base-german-cased] cp to hardcoded urls (#10353) 2021-02-23 12:30:47 -05:00
Sylvain Gugger
461e8cacf9 Fix evaluation with label smoothing in Trainer (#10338) 2021-02-22 16:39:02 -05:00
Stas Bekman
622a8c5995 [trainer] add Trainer methods for metrics logging and saving (#10266) 
* make logging and saving trainer built-in

* Update src/transformers/trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-02-22 13:02:53 -08:00
Tanmay Garg
94d8767ba3 Loading from last checkpoint functionality in Trainer.train (#10334) 
Enhance resume_from_checkpoint argument of Trainer.train to accept
bool type. If True given, last saved checkpoint in self.args.output_dir
will be loaded. (#10280)
 2021-02-22 15:33:00 -05:00
Stas Bekman
eab0afc19c [Trainer] implement gradient_accumulation_steps support in DeepSpeed integration (#10310) 
* implement gradient_accumulation_steps support in DeepSpeed integration

* typo

* cleanup

* cleanup
 2021-02-22 11:15:59 -08:00
Sylvain Gugger
9e147d31f6 Deprecate prepare_seq2seq_batch (#10287) 
* Deprecate prepare_seq2seq_batch

* Fix last tests

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* More review comments

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Suraj Patil <surajp815@gmail.com>
 2021-02-22 12:36:16 -05:00
Lysandre Debut
e73a3e1891 Add note to resize token embeddings matrix when adding new tokens to voc (#10331) 2021-02-22 09:48:20 -05:00
Julien Plu
19e737b93e Making TF Longformer-like models compliant with AMP (#10233) 
* AMP

* Add LED

* Apply style

* Fix longformer
 2021-02-22 15:41:56 +01:00
tagucci
88605f37a6 fix typo in conversion script (#10316) 
* fix typo in conversion script

* style

Co-authored-by: Stas Bekman <stas@stason.org>
 2021-02-21 07:54:27 -08:00
Sylvain Gugger
a2e379743c Fix style 2021-02-20 15:46:54 -05:00
cronoik
a0dfc2d30f fixes #10303 (#10304) 2021-02-20 15:21:33 -05:00
Pengcheng He
9a7e63729f Integrate DeBERTa v2(the 1.5B model surpassed human performance on Su… (#10018) 
* Integrate DeBERTa v2(the 1.5B model surpassed human performance on SuperGLUE); Add DeBERTa v2 900M,1.5B models;

* DeBERTa-v2

* Fix v2 model loading issue (#10129)

* Doc members

* Update src/transformers/models/deberta/modeling_deberta.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Address Sylvain's comments

* Address Patrick's comments

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Style

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2021-02-19 18:34:44 -05:00
Tanmay Garg
709c86b5a9 Introduce logging_strategy training argument (#10267) (#10267) 
Introduce logging_strategy training argument
in TrainingArguments and TFTrainingArguments. (#9838)
 2021-02-19 11:49:22 -05:00
Julien Plu
34df26ec3a Making TF OpenAI GPT model compliant with AMP and XLA (#10261) 
* Fix AMP and XLA

* Remove useless var
 2021-02-19 09:33:25 -05:00
Julien Plu
3e116ed331 Making TF TransfoXL model compliant with AMP (#10264) 
* Fix AMP

* Apply style

* Remove unused import
 2021-02-19 06:58:07 -05:00
Julien Plu
86caeb7636 Fix XLA and AMP (#10262) 2021-02-19 06:57:16 -05:00
Julien Plu
3d72d47f09 Making TF MPNet model compliant with XLA (#10260) 
* Fix XLA

* Rework cast

* Apply style
 2021-02-19 06:56:41 -05:00
Julien Plu
fb56bf2584 Making TF MobileBert model compliant with AMP (#10259) 
* Fix AMP

* Trigger CI

* Rework cast
 2021-02-19 06:55:25 -05:00
Julien Plu
2fc6284f04 Making TF Lxmert model compliant with AMP (#10257) 
* Fix AMP

* Rework cast

* Apply style
 2021-02-19 06:54:14 -05:00
Stas Bekman
4eddc459a9 [trainer] implement support for full fp16 in evaluation/predict (#10268) 
* implement --fp16_full_eval

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* style

* add test

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-02-18 17:02:35 -08:00
Stas Bekman
97e688bc22 [Trainer] memory tracker metrics (#10225) 
* memory tracker metrics

* go back to eval for somewhat consistency

* handle no-gpu case

* deal with stackable eval calls

* restore callback order

* style

* simplify the API

* add test

* docs

* consistently use eval_ prefix

* improve docs

* Update src/transformers/trainer_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* rename method

* style

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-02-18 09:27:32 -08:00
Tanmay Garg
d7f38c5d1d Introduce warmup_ratio training argument (#10229) 
Introduce warmup_ratio training argument in both
TrainingArguments and TFTrainingArguments classes (#6673)
 2021-02-18 12:23:33 -05:00
Julien Plu
2acae50a0c Reduce the time spent for the TF slow tests (#10152) 
* rework savedmodel slow test

* Improve savedmodel tests

* Remove useless content
 2021-02-18 15:52:57 +01:00
Julien Plu
14ed3b978e Fix AMP (#10216) 2021-02-18 06:29:43 -05:00
Julien Plu
bdf1669e3f Making TF GPT2 compliant with XLA and AMP (#10230) 
* Fix XLA and AMP

* Fix AMP and XLA

* Apply style

* Apply Patrick's comment
 2021-02-18 09:36:01 +01:00
Stas Bekman
dee876ceff [trainer] refactor place_model_on_device logic, add deepspeed (#10243) 
* refactor place_model_on_device logic, add deepspeed

* doc

* style
 2021-02-17 15:52:36 -08:00
Julien Plu
7246785a67 Make TF CTRL compliant with XLA and AMP (#10209) 
* Fix XLA and AMP

* Apply style

* Remove useless cast
 2021-02-17 18:54:15 +01:00
Julien Plu
fdb2351ebb Making TF XLM-like models XLA and AMP compliant (#10211) 
* Fix Flaubert and XLM

* Remove useless cast

* Tiny fix

* Tiny fix
 2021-02-17 18:02:48 +01:00
Julien Plu
83d803ba02 Making TF BART-like models XLA and AMP compliant (#10191) 
* Update BART

* Update Blenderbot

* Update BlenderbotSmall

* Update Marian

* Update MBart

* Update MBart

* Update Pegasus

* Update template

* Fix Marian and Pegasus

* Apply style

* Default initializer

* Default initializer

* Default initializer

* Remove int32 casts

* Fix template

* Remove more cast
 2021-02-17 17:48:56 +01:00
Daniel Stancl
8d79e5ca49 Fix head masking for TFT5 (#9877) 
* Fix head_mask and decoder_head_mask in TFT5 models

* Enable test_headmasking both fot TFT5 tester
and TFT5EncoderOnly tester

Co-authored-by: patrickvonplaten <patrick.v.platen@gmail.com>
 2021-02-17 19:00:09 +03:00
Lysandre Debut
4b91965731 Factor out methods (#10215) 2021-02-17 09:53:43 -05:00
Stas Bekman
e94d63f6cb [trainer] fix ignored columns logger (#10219) 
* [trainer] fix ignored columns logger

This PR fixes a confusing log entry that says:
```
The following columns in the evaluation set don't have a corresponding argument in `T5ForConditionalGeneration.forward` and have been ignored: .
```
when everything is in order.

* Update src/transformers/trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-02-16 13:35:39 -08:00