SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
b4d554880013bf97718e1e1332715eeaba7dee17
HuggingFace_transformer/tests/models
History
Arthur b4d5548800 🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224) 
* fix EVERYTHING

* more fixes

* ⚗️⚗️ Tokenizer magic ⚗️⚗️

* wrong value but test passes for the TODO

* update

* updat

* safe protobuf import?

* style

* non gated repo

* update

* fixup

* Update src/transformers/models/llama/tokenization_llama.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update src/transformers/models/llama/tokenization_llama.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update tests/models/t5/test_tokenization_t5.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* nits

* fix t5 too

* use assert equal

* fix llama decoding

* nits on t5

* fixup

* only remove the prefix space, not other spaces

* more deconding tests and more todos

* fix CI as well

* fixup

* skip failing test on CI (its tf its ok)

* skip test_subword_regularization_tokenizer that is also crashing on the CI for TF

* update llama

* revert good fixes

* fixup

* empty

* explain why we need to encode with an additional token

* better warning?

* nits

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-17 17:08:05 +02:00
..
albert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
align
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
altclip
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
audio_spectrogram_transformer
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
auto
Remote code improvements (#23959)
2023-06-06 14:31:14 -04:00
autoformer
Compute dropout_probability only in training mode (#24486)
2023-06-26 18:36:47 +02:00
bark
Update Bark generation configs and tests (#25409)
2023-08-09 18:28:02 +02:00
bart
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
barthez
…
bartpho
…
beit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
bert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
bert_generation
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
bert_japanese
…
bertweet
…
big_bird
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
bigbird_pegasus
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
biogpt
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
bit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
blenderbot
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
blenderbot_small
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
blip
Input data format (#25464)
2023-08-16 17:45:02 +01:00
blip_2
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
bloom
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
bridgetower
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
byt5
…
camembert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
canine
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
chinese_clip
Input data format (#25464)
2023-08-16 17:45:02 +01:00
clap
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
clip
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
clipseg
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
codegen
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
conditional_detr
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
convbert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
convnext
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
convnextv2
Update ConvNextV2ModelIntegrationTest::test_inference_image_classification_head (#23402)
2023-05-16 23:35:11 +02:00
cpm
Fix PipelineTests skip conditions (#22320)
2023-03-22 20:02:24 +01:00
cpmant
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
ctrl
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
cvt
Make more test models smaller (#25005)
2023-07-24 10:08:47 -04:00
data2vec
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
deberta
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
deberta_v2
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
decision_transformer
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
deformable_detr
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
deit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
deta
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
detr
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
dinat
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
dinov2
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
distilbert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
dit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
donut
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
dpr
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
dpt
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
efficientformer
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
efficientnet
🚨🚨🚨 Remove softmax for EfficientNetForImageClassification 🚨🚨🚨 (#25501)
2023-08-14 17:08:47 +01:00
electra
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
encodec
Make more test models smaller (#25005)
2023-07-24 10:08:47 -04:00
encoder_decoder
Move TF building to an actual build() method (#23760)
2023-06-06 18:30:51 +01:00
ernie
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
ernie_m
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
esm
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
falcon
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
flaubert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
flava
Input data format (#25464)
2023-08-16 17:45:02 +01:00
fnet
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
focalnet
Update tiny models and pipeline tests (#23446)
2023-05-18 17:29:04 +02:00
fsmt
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
funnel
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
git
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
glpn
Input data format (#25464)
2023-08-16 17:45:02 +01:00
gpt2
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gpt_bigcode
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gpt_neo
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gpt_neox
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gpt_neox_japanese
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gpt_sw3
…
gptj
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
gptsan_japanese
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
graphormer
Make more test models smaller (#25005)
2023-07-24 10:08:47 -04:00
groupvit
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
herbert
…
hubert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
ibert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
imagegpt
Input data format (#25464)
2023-08-16 17:45:02 +01:00
informer
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
instructblip
Update InstructBLIP & Align values after rescale update (#25209)
2023-08-03 11:01:10 +01:00
jukebox
Set TF32 flag for PyTorch cuDNN backend (#25075)
2023-07-25 08:04:48 -04:00
layoutlm
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
layoutlmv2
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
layoutlmv3
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
layoutxlm
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
led
Speed up TF tests by reducing hidden layer counts (#24595)
2023-06-30 16:30:33 +01:00
levit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
lilt
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
llama
🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224)
2023-08-17 17:08:05 +02:00
longformer
Fix more offload edge cases (#25342)
2023-08-07 17:45:41 +02:00
longt5
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
luke
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
lxmert
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
m2m_100
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
marian
Marian: post-hack-fix correction (#25459)
2023-08-16 11:49:29 +01:00
markuplm
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
mask2former
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
maskformer
Fix MaskFormerModelIntegrationTest OOM (#25544)
2023-08-16 18:11:24 +02:00
mbart
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
mbart50
…
mega
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
megatron_bert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
megatron_gpt2
…
mgp_str
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
mluke
…
mobilebert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
mobilenet_v1
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
mobilenet_v2
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
mobilevit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
mobilevitv2
Make more test models smaller (#25005)
2023-07-24 10:08:47 -04:00
mpnet
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
mpt
Fix MPT CI (#25548)
2023-08-17 09:06:26 +02:00
mra
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
mt5
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
musicgen
[MusicGen] Fix integration tests (#25169)
2023-07-28 18:50:15 +01:00
mvp
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
nat
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
nezha
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
nllb
🚨🚨🚨 [NLLB Tokenizer] Fix the prefix tokens 🚨🚨🚨 (#22313)
2023-04-04 14:53:06 +02:00
nllb_moe
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
nystromformer
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
oneformer
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
openai
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
opt
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
owlvit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
pegasus
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
pegasus_x
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
perceiver
Fix last models for common tests that are too big. (#25058)
2023-07-25 07:56:04 -04:00
phobert
…
pix2struct
Input data format (#25464)
2023-08-16 17:45:02 +01:00
plbart
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
poolformer
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
prophetnet
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
pvt
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
qdqbert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
rag
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
realm
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
reformer
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
regnet
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
rembert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
resnet
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
roberta
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
roberta_prelayernorm
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
roc_bert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
roformer
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
rwkv
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
sam
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
segformer
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
sew
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
sew_d
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
speech_encoder_decoder
…
speech_to_text
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
speech_to_text_2
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
speecht5
add generate method to SpeechT5ForTextToSpeech (#25233)
2023-08-03 14:12:07 +01:00
splinter
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
squeezebert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
swiftformer
Fix last models for common tests that are too big. (#25058)
2023-07-25 07:56:04 -04:00
swin
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
swin2sr
Input data format (#25464)
2023-08-16 17:45:02 +01:00
swinv2
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
switch_transformers
Switch Transformers: remove overwritten beam sample test (#25458)
2023-08-11 13:16:01 +01:00
t5
🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224)
2023-08-17 17:08:05 +02:00
table_transformer
Fix last models for common tests that are too big. (#25058)
2023-07-25 07:56:04 -04:00
tapas
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
time_series_transformer
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
timesformer
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
timm_backbone
Fix last models for common tests that are too big. (#25058)
2023-07-25 07:56:04 -04:00
transfo_xl
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
trocr
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
tvlt
Input data format (#25464)
2023-08-16 17:45:02 +01:00
umt5
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
unispeech
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
unispeech_sat
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
upernet
Fix last models for common tests that are too big. (#25058)
2023-07-25 07:56:04 -04:00
videomae
Input data format (#25464)
2023-08-16 17:45:02 +01:00
vilt
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
vision_encoder_decoder
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vision_text_dual_encoder
Fix VisionTextDualEncoderIntegrationTest (#24661)
2023-07-05 13:44:30 +02:00
visual_bert
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
vit
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
vit_hybrid
fix vit hybrid test (#25543)
2023-08-16 17:02:57 +02:00
vit_mae
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
vit_msn
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
vivit
Input data format (#25464)
2023-08-16 17:45:02 +01:00
wav2vec2
Mark flaky tests (#25463)
2023-08-11 15:26:45 +01:00
wav2vec2_conformer
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
wav2vec2_phoneme
…
wav2vec2_with_lm
Fix test_word_time_stamp_integration for Wav2Vec2ProcessorWithLMTest (#22800)
2023-04-17 12:41:55 +02:00
wavlm
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
whisper
Skip test_beam_search_xla_generate_simple for T5 (#25566)
2023-08-17 15:30:46 +02:00
x_clip
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
xglm
Fix test_model_parallelism (#25359)
2023-08-08 10:48:45 +02:00
xlm
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
xlm_prophetnet
Update expected values in XLMProphetNetModelIntegrationTest (#21957)
2023-03-06 09:15:44 +01:00
xlm_roberta
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
xlm_roberta_xl
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
xlnet
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
xmod
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
yolos
Refactor image processor testers (#25450)
2023-08-11 11:30:18 +01:00
yoso
CI with num_hidden_layers=2 🚀🚀🚀 (#25266)
2023-08-02 20:22:36 +02:00
__init__.py
…