SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
8,984 Commits 5 Branches 239 Tags
2e12b907ae73edd2f3590cefdfa5d11f74aabfac
Commit Graph

8984 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Patrick von Platen
2e12b907ae TF generate refactor - Greedy Search (#15562) 
* TF generate start refactor

* Add tf tests for sample generate

* re-organize

* boom boom

* Apply suggestions from code review

* re-add

* add all code

* make random greedy pass

* make encoder-decoder random work

* further improvements

* delete bogus file

* make gpt2 and t5 tests work

* finish logits tests

* correct logits processors

* correct past / encoder_outputs drama

* refactor some methods

* another fix

* refactor shape_list

* fix more shape list

* import shape
_list

* finish docs

* fix imports

* make style

* correct tf utils

* Fix TFRag as well

* Apply Lysandre's and Sylvais suggestions

* Update tests/test_generation_tf_logits_process.py

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

* Update src/transformers/tf_utils.py

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

* remove cpu according to gante

* correct logit processor

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
 2022-02-15 17:54:43 +01:00
Nicolas Patry
a3dbbc3467 Add decoder_kwargs to send to LM on asr pipeline. (#15646) 
Co-authored-by: Giuseppe Attanasio <giuseppeattanasio6@gmail.com>

Co-authored-by: Giuseppe Attanasio <giuseppeattanasio6@gmail.com>
 2022-02-15 17:53:24 +01:00
Nicolas Patry
cdf19c501d Re-export KeyDataset. (#15645) 
* Re-export `KeyDataset`.

* Update the docs locations.
 2022-02-15 17:49:38 +01:00
Stas Bekman
28e6155d8a add a network debug script and document it (#15652) 
* add a network debug script and document it

* doc
 2022-02-15 08:48:00 -08:00
Sylvain Gugger
5d8be090e0 Fix quality 2022-02-15 11:32:26 -05:00
Patrick von Platen
f45ac11fb3 Add section about doc testing (#15659) 
* Add doctesting section

* Improve

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-02-15 16:56:31 +01:00
Shamane Siri
80f1a59168 updated with latest PL and Ray (#15653) 2022-02-15 16:53:05 +01:00
Ngo Quang Huy
7bc4a01cb5 Update bad_words_ids usage (#15641) 
* Improve the parameter `bad_word_ids' usage

* Update the bad_words_ids strategy
 2022-02-15 16:44:34 +01:00
arampacha
67047b86ce add scores to Wav2Vec2WithLMOutput (#15413) 
* add scores to Wav2Vec2WithLMOutput

* style fixup
 2022-02-15 16:40:50 +01:00
Sylvain Gugger
45f56580a7 Allow custom code for Processors (#15649) 
* Allow custom code for Processors

* Add more test

* Test all auto_map configs are properly set
 2022-02-15 09:44:35 -05:00
jonrbates
86a7845c0c Fix typo in speech2text2 doc (#15617) 
Forward looks for inputs, not input_ids
 2022-02-15 13:54:34 +01:00
Javier de la Rosa
9eb7e9ba1d Fix ASR pipelines from local directories with wav2vec models that have language models attached (#15590) 
* Fix loading pipelines with wav2vec models with lm when in local paths

* Adding tests

* Fix test

* Adding tests

* Flake8 fixes

* Removing conflict files :(

* Adding task type to test

* Remove unnecessary test and imports
 2022-02-15 13:45:08 +01:00
Alex Hedges
e1cbc073bf Require tokenizers>=0.11.1 (#15266) 
`tokenizers` version that supports the feature to choose the direction of truncation
 2022-02-15 11:46:12 +01:00
fra
05a8580964 Revert "logger doc" 
This reverts commit 41168a49ce.
 2022-02-15 10:46:45 +01:00
fra
41168a49ce logger doc 2022-02-15 10:03:28 +01:00
Patrick von Platen
041fdc4a7e [SpeechEncoderDecoder] Make sure no EOS is generated in test (#15655) 2022-02-15 09:13:55 +01:00
muzhi1991
e314c19a3f fix bug for the log of RNG states are not properly loaded exception. (#15638) 
Co-authored-by: muz <muzhi1991@limuzhideMBP-2.lan>
 2022-02-14 20:30:55 -05:00
Sylvain Gugger
2e11a04337 Register feature extractor (#15634) 
* Rework AutoFeatureExtractor.from_pretrained internal

* Custom feature extractor

* Add more tests

* Add support for custom feature extractor code

* Clean up

* Add register API to AutoFeatureExtractor
 2022-02-14 13:35:16 -05:00
lewtun
0f71c29053 Remove redundant error logging in from_pretrained() method (#15631) 
* Remove error logging in from_pretrained() method
 2022-02-14 18:03:07 +01:00
NielsRogge
b090b79022 Make Swin work with VisionEncoderDecoderModel (#15527) 
* Add attribute_map

* Add mention in docs

* Set hidden_size attribute correctly

* Add note about Transformer-based models only

Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>
 2022-02-14 17:33:35 +01:00
Toni Kukurin
ec15da2445 Report only the failed imports in requires_backends (#15636) 2022-02-14 10:35:20 -05:00
Zhen Wang
2b8599b2df Fix a bug that ignores max_seq_len in preprocess (#15238) 2022-02-14 13:18:40 +01:00
Yih-Dar
f52746d004 [Fix doc example] FlaxVisionEncoderDecoder (#15626) 
* Fix wrong checkpoint name: vit

* Fix missing import

* Fix more missing import

* make style

* Apply suggestions from code review

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
 2022-02-14 12:48:23 +01:00
Sylvain Gugger
52d2e6f6e9 Add push to hub to feature extractor (#15632) 
* Add push to hub to feature extractor

* Quality

* Clean up
 2022-02-11 17:14:01 -05:00
Daniel Erenrich
4f403ea899 Fix grammar in tokenizer_summary (#15614) 
"to make ensure" is redundant.
 2022-02-11 16:51:30 -05:00
Sylvain Gugger
7a32e4722f Custom feature extractor (#15630) 
* Rework AutoFeatureExtractor.from_pretrained internal

* Custom feature extractor

* Add more tests

* Add support for custom feature extractor code

* Clean up
 2022-02-11 16:43:54 -05:00
Stas Bekman
fcb0f74397 [research_projects] deal with security alerts (#15594) 
* [research_projects] deal with security alerts

* add a note of the original PL ver and warning
 2022-02-11 14:31:09 -05:00
Stas Bekman
f15c99fabf [deepspeed docs] misc additions (#15585) 
* [deepspeed docs] round_robin_gradients

* training and/or eval/predict loss is

* Update docs/source/main_classes/deepspeed.mdx

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-02-11 10:54:04 -08:00
Sylvain Gugger
2dce350b33 Fix _configuration_file argument getting passed to model (#15629) 2022-02-11 13:46:08 -05:00
Steven Liu
85aee09e9a 🖍 remove broken link (#15615) 2022-02-11 12:33:55 -06:00
Joao Gante
2f40c728c9 TF MT5 embeddings resize (#15567) 
* Fix TF MT5 vocab resize

* more assertive testing
 2022-02-11 17:35:10 +00:00
Mishig Davaadorj
8c03df1010 Rebase (#15606) 2022-02-11 12:02:02 -05:00
Joao Gante
3fae83d23a TF: Add informative warning for inexistent CPU backprop ops (#15612) 
* Add informative warning
 2022-02-11 16:16:26 +00:00
lewtun
7e4844fc2a Enable ONNX export when PyTorch and TensorFlow installed in the same environment (#15625) 2022-02-11 16:25:06 +01:00
Sylvain Gugger
6cf06d198c Mark "code in the Hub" API as experimental (#15624) 2022-02-11 09:55:31 -05:00
Patrick von Platen
45c7b5b1c7 [Generate] Small refactor (#15611) 2022-02-10 18:29:27 +01:00
Ngo Quang Huy
c0864d98ba Correct JSON format (#15600) 2022-02-10 09:02:03 -08:00
lewtun
2e8b85f72e Add local and TensorFlow ONNX export examples to docs (#15604) 
* Add local and TensorFlow ONNX export examples to docs

* Use PyTorch - TensorFlow split
 2022-02-10 16:31:00 +01:00
NielsRogge
3a2ed96714 Fix Seq2SeqTrainer (#15603) 
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>
 2022-02-10 16:26:14 +01:00
Yih-Dar
724e51c6e6 Compute loss independent from decoder for TF EncDec models (as #14139) (#15175) 
* Compute loss independent from decoder (as 14139)

* fix expected seq_len + style

* Apply the same change to TFVisionEncoderDecoderModel

* fix style

* Add case with labels in equivalence test

* uncomment

* Add case with labels in equivalence test

* add decoder_token_labels

* use hf_compute_loss

* Apply suggestions from code review

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* Add copied from

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
 2022-02-10 15:47:02 +01:00
Patrick von Platen
3d5dea9bf0 Add example batch size to all commands (#15596) 2022-02-10 08:52:07 -05:00
Alberto Bégué
cb7ed6e083 Add Tensorflow handling of ONNX conversion (#13831) 
* Add TensorFlow support for ONNX export

* Change documentation to mention conversion with Tensorflow

* Refactor export into export_pytorch and export_tensorflow

* Check model's type instead of framework installation to choose between TF and Pytorch

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Alberto Bégué <alberto.begue@della.ai>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
 2022-02-10 11:18:41 +01:00
Lysandre
e923917cd9 Reformat tokenization_fnet 2022-02-09 22:23:32 -05:00
Sylvain Gugger
644ec05233 Make slow tests slow 2022-02-09 19:10:22 -05:00
Sylvain Gugger
c722753afd Expand tutorial for custom models (#15587) 
* Expand tutorial for custom models

* Style

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
 2022-02-09 17:44:28 -05:00
NielsRogge
a86ee2261e Add link (#15588) 
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MBP.localdomain>
 2022-02-09 23:33:39 +01:00
Stas Bekman
dee17d5676 [trainer docs] document how to select specific gpus (#15551) 
* [trainer docs] document how to select specific gpus

* expand

* add urls

* add accelerate launcher
 2022-02-09 10:12:29 -08:00
Yih-Dar
258480864d update serving_output for some TF models (#15568) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-02-09 18:32:51 +01:00
Sylvain Gugger
315e67404d Fix tests hub failure (#15580) 
* Expose hub test problem

* Fix tests
 2022-02-09 12:27:59 -05:00
Sylvain Gugger
b1ba03e082 Fix quality 2022-02-09 12:06:59 -05:00