SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
2,449 Commits 5 Branches 239 Tags
18e1f751f1d996c4fe01559ade1cd013186b81e4
Commit Graph

381 Commits

Author SHA1 Message Date
Julien Chaumond
18e1f751f1 TF support 2019-12-11 17:07:46 -05:00
Julien Chaumond
31e5b5ff22 Fix tests + first example of doc 2019-12-11 15:22:02 -05:00
Julien Chaumond
c999a3e505 Allow from_pretrained to take a remote identifier 2019-12-11 12:29:58 -05:00
thomwolf
29570db25b allowing from_pretrained to load from url directly 2019-12-11 17:19:18 +01:00
Julien Chaumond
2e2f9fed55 rm duplicate imports 2019-12-11 11:11:56 -05:00
LysandreJik
4c12860f7a Remove misleading documentation 2019-12-11 09:22:37 -05:00
Thomas Wolf
51ae203290 Merge pull request #2129 from leopd/master 
Progress indicator improvements when downloading pre-trained models.
 2019-12-10 22:18:55 +01:00
Leo Dirac
58d75aa310 Progress indicator improvements when downloading pre-trained models. 2019-12-10 11:36:56 -08:00
LysandreJik
6a73382706 Complete warning + cleanup 2019-12-10 14:33:24 -05:00
Lysandre
dc4e9e5cb3 DataParallel for SQuAD + fix XLM 2019-12-10 19:21:20 +00:00
Thomas Wolf
e6cff60b4c Merge pull request #2069 from huggingface/cleaner-pt-tf-conversion 
clean up PT <=> TF conversion
 2019-12-10 15:34:08 +01:00
Thomas Wolf
e57d00ee10 Merge pull request #1984 from huggingface/squad-refactor 
[WIP] Squad refactor
 2019-12-10 11:07:26 +01:00
Thomas Wolf
ecabbf6d28 Merge pull request #2107 from huggingface/encoder-mask-shape 
create encoder attention mask from shape of hidden states
 2019-12-10 10:07:56 +01:00
Rémi Louf
f7eba09007 clean for release 2019-12-09 20:37:55 -05:00
Rémi Louf
c0443df593 remove beam search 2019-12-09 20:37:55 -05:00
Rémi Louf
2403a66598 give transformers API to BertAbs 2019-12-09 20:37:55 -05:00
Rémi Louf
4d18199902 cast bool tensor to long for pytorch < 1.3 2019-12-09 20:37:55 -05:00
Rémi Louf
9f75565ea8 setup training 2019-12-09 20:37:55 -05:00
Rémi Louf
4735c2af07 tweaks to the BeamSearch API 2019-12-09 20:37:55 -05:00
Rémi Louf
ba089c780b share pretrained embeddings 2019-12-09 20:37:55 -05:00
Rémi Louf
9660ba1cbd Add beam search 2019-12-09 20:37:55 -05:00
Rémi Louf
1c71ecc880 load the pretrained weights for encoder-decoder 
We currently save the pretrained_weights of the encoder and decoder in
two separate directories `encoder` and `decoder`. However, for the
`from_pretrained` function to operate with automodels we need to
specify the type of model in the path to the weights.

The path to the encoder/decoder weights is handled by the
`PreTrainedEncoderDecoder` class in the `save_pretrained` function. Sice
there is no easy way to infer the type of model that was initialized for
the encoder and decoder we add a parameter `model_type` to the function.
This is not an ideal solution as it is error prone, and the model type
should be carried by the Model classes somehow.

This is a temporary fix that should be changed before merging.
 2019-12-09 20:37:55 -05:00
Lysandre Debut
00c4e39581 Merge branch 'master' into squad-refactor 2019-12-09 10:41:15 -05:00
Rémi Louf
3520be7824 create encoder attention mask from shape of hidden states 
We currently create encoder attention masks (when they're not provided)
based on the shape of the inputs to the encoder. This is obviously
wrong; sequences can be of different lengths. We now create the encoder
attention mask based on the batch_size and sequence_length of the
encoder hidden states.
 2019-12-09 11:19:45 +01:00
Aymeric Augustin
0cb163865a Remove pytest dependency. (#2093) 2019-12-07 07:46:14 -05:00
Michael Watkins
2670b0d682 Fix bug which lowercases special tokens 2019-12-06 16:15:53 -05:00
Aymeric Augustin
35401fe50f Remove dependency on pytest for running tests (#2055) 
* Switch to plain unittest for skipping slow tests.

Add a RUN_SLOW environment variable for running them.

* Switch to plain unittest for PyTorch dependency.

* Switch to plain unittest for TensorFlow dependency.

* Avoid leaking open files in the test suite.

This prevents spurious warnings when running tests.

* Fix unicode warning on Python 2 when running tests.

The warning was:

    UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

* Support running PyTorch tests on a GPU.

Reverts 27e015bd.

* Tests no longer require pytest.

* Make tests pass on cuda
 2019-12-06 13:57:38 -05:00
Julien Chaumond
e4679cddce [cli] Uploads: add progress bar (#2078) 
* [cli] Uploads: add progress bar

see https://github.com/huggingface/transformers/pull/2044#discussion_r354057827 for context

* rename + documentation

* Add auto-referential comment
 2019-12-06 11:56:23 -05:00
thomwolf
1d87b37d10 updating 2019-12-06 15:30:09 +01:00
Thomas Wolf
4cb9b60558 Merge pull request #2077 from patrickvonplaten/change_documentation_for_past_output_shape 
corrected documentation for past tensor shape for ctrl and gpt2 model
 2019-12-06 12:14:48 +01:00
Thomas Wolf
5482822a2b Merge pull request #2046 from jplu/tf2-ner-example 
Add NER TF2 example.
 2019-12-06 12:12:22 +01:00
Thomas Wolf
fc1bb1f867 Merge pull request #2068 from huggingface/fix-2042 
Nicer error message when Bert's input is missing batch size
 2019-12-06 12:06:42 +01:00
patrickvonplaten
d0383e4daf corrected documentation for past tensor shape for ctrl and gpt2 model 2019-12-06 01:24:22 +01:00
LysandreJik
e9217da5ff Cleanup 
Improve global visibility on the run_squad script, remove unused files and fixes related to XLNet.
 2019-12-05 16:01:51 -05:00
LysandreJik
9ecd83dace Patch evaluation for impossible values + cleanup 2019-12-05 14:44:57 -05:00
thomwolf
f8fb4335c9 clean up a little bit PT <=> TF conversion 2019-12-05 15:19:32 +01:00
Thomas Wolf
bebaa14039 Merge pull request #2045 from aaugustin/remove-dead-code 
Remove dead code in tests.
 2019-12-05 14:41:56 +01:00
thomwolf
18fb93530b fixing #2042 - Nicer error message 2019-12-05 14:36:34 +01:00
thomwolf
2d5d86e037 fix #2031 2019-12-05 14:06:29 +01:00
thomwolf
3268ebd229 fix xlnet test 2019-12-05 13:35:29 +01:00
thomwolf
6c5297a423 Fixing camembert tokenization 2019-12-05 13:27:58 +01:00
Julien Plu
9200a759d7 Add few tests on the TF optimization file with some info in the documentation. Complete the README. 2019-12-05 12:56:43 +01:00
Thomas Wolf
1eaf44e713 Merge pull request #2007 from roskoN/xlnet_attention_fix 
fixed XLNet attention output for both attention streams whenever target_mapping is provided
 2019-12-05 12:32:39 +01:00
thomwolf
71e4693f08 fix #1968 2019-12-05 12:14:24 +01:00
Thomas Wolf
f9f395b21c Merge pull request #1735 from ondewo/tf-do-not-use-gpu-on-import 
Do not use GPU when importing transformers
 2019-12-05 11:56:48 +01:00
thomwolf
8b388827b5 fix #1920 2019-12-05 11:18:43 +01:00
Thomas Wolf
d425a4d60b Merge pull request #1870 from alexzubiaga/xlnet-for-token-classification 
XLNet for Token classification
 2019-12-05 09:54:09 +01:00
Thomas Wolf
1eb89ddf73 Merge pull request #2044 from huggingface/cli_upload 
CLI for authenticated file sharing
 2019-12-05 09:44:07 +01:00
VictorSanh
fb0d2f1da1 preparing release distil-mBERT 2019-12-05 03:00:16 -05:00
Julien Chaumond
3ba417e1a8 [cli] ls: Tabular formatting 2019-12-04 18:40:52 -05:00