SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity
Files
b52a03cd3bec92d0ee84f0b1f7edee0d5117200a
HuggingFace_transformer/tests/models
History
Arthur b52a03cd3b ⚠️⚠️[T5Tokenize] Fix T5 family tokenizers⚠️⚠️ (#24565) 
* don't add space before single letter chars that don't have a merge

* fix the fix

* fixup

* add a test

* more testing

* fixup

* hack to make sure fast is also fixed

* update switch transformers test

* revert convert slow

* Update src/transformers/models/t5/tokenization_t5.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* add typechecking

* quality

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2023-06-30 07:00:43 +02:00
..
albert
TF: standardize test_model_common_attributes for language models (#23457)
2023-06-13 17:51:37 +01:00
align
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
altclip
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
audio_spectrogram_transformer
is_batched fix for remaining 2-D numpy arrays (#23309)
2023-05-23 14:37:35 -04:00
auto
Remote code improvements (#23959)
2023-06-06 14:31:14 -04:00
autoformer
Compute dropout_probability only in training mode (#24486)
2023-06-26 18:36:47 +02:00
bart
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
barthez
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
bartpho
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
beit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
bert
TF: standardize test_model_common_attributes for language models (#23457)
2023-06-13 17:51:37 +01:00
bert_generation
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
bert_japanese
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
bertweet
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
big_bird
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
bigbird_pegasus
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
biogpt
Update tiny models and pipeline tests (#23446)
2023-05-18 17:29:04 +02:00
bit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
blenderbot
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
blenderbot_small
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
blip
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
blip_2
Add InstructBLIP (#23460)
2023-06-26 11:23:57 +02:00
bloom
Byebye pytorch 1.9 (#24080)
2023-06-16 16:38:23 +02:00
bort
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
bridgetower
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
byt5
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
camembert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
canine
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
chinese_clip
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
clap
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
clip
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
clipseg
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
codegen
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
conditional_detr
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
convbert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
convnext
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
convnextv2
Update ConvNextV2ModelIntegrationTest::test_inference_image_classification_head (#23402)
2023-05-16 23:35:11 +02:00
cpm
Fix PipelineTests skip conditions (#22320)
2023-03-22 20:02:24 +01:00
cpmant
Update tiny models and pipeline tests (#23446)
2023-05-18 17:29:04 +02:00
ctrl
TF: CTRL with native embedding layers (#23456)
2023-06-14 14:39:02 +01:00
cvt
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
data2vec
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
deberta
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
deberta_v2
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
decision_transformer
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
deformable_detr
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
deit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
deta
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
detr
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
dinat
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
distilbert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
dit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
donut
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
dpr
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
dpt
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
efficientformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
efficientnet
Update tiny model summary file for recent models (#22637)
2023-04-06 22:52:59 +02:00
electra
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
encodec
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
encoder_decoder
Move TF building to an actual build() method (#23760)
2023-06-06 18:30:51 +01:00
ernie
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
ernie_m
Automatically create/update tiny models (#22275)
2023-03-23 19:14:17 +01:00
esm
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
flaubert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
flava
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
fnet
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
focalnet
Update tiny models and pipeline tests (#23446)
2023-05-18 17:29:04 +02:00
fsmt
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
funnel
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
git
Add a check in ImageToTextPipeline._forward (#24373)
2023-06-20 18:07:34 +02:00
glpn
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
gpt2
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
gpt_bigcode
Add torch >=1.12 requirement for Tapas (#24251)
2023-06-13 19:19:40 +02:00
gpt_neo
Update tiny models and pipeline tests (#23446)
2023-05-18 17:29:04 +02:00
gpt_neox
[GPTNeoX] Nit in config (#24349)
2023-06-20 19:19:19 +02:00
gpt_neox_japanese
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
gpt_sw3
Add gpt-sw3 model to transformers (#20209)
2022-12-12 13:12:13 -05:00
gptj
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
gptsan_japanese
Update tiny model summary file for recent models (#22637)
2023-04-06 22:52:59 +02:00
graphormer
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
groupvit
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
herbert
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
hubert
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
ibert
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
imagegpt
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
informer
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
instructblip
[InstructBlip] Add instruct blip int8 test (#24555)
2023-06-28 19:06:30 +02:00
jukebox
Update Jukebox tests (#21984)
2023-03-07 04:20:14 +01:00
layoutlm
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
layoutlmv2
Update tiny models for pipeline testing. (#24364)
2023-06-20 14:43:10 +02:00
layoutlmv3
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
layoutxlm
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
led
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
levit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
lilt
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
llama
[LlamaTokenizerFast] nit update post_processor on the fly (#23855)
2023-05-30 16:50:41 +02:00
longformer
TensorFlow CI fixes (#24360)
2023-06-20 12:59:21 +01:00
longt5
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
luke
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
lxmert
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
m2m_100
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
marian
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
markuplm
Update some MarkupLM tests' expected values (#22667)
2023-04-11 10:00:34 +02:00
mask2former
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
maskformer
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
mbart
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
mbart50
Replace as_target context managers by direct calls (#18325)
2022-07-29 08:09:09 -04:00
mctct
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
mega
Fix MegaModel CI (#22652)
2023-04-07 17:13:04 +02:00
megatron_bert
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
megatron_gpt2
Move test model folders (#17034)
2022-05-03 14:42:02 +02:00
mgp_str
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
mluke
Black preview (#17217)
2022-05-12 16:25:55 -04:00
mobilebert
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
mobilenet_v1
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
mobilenet_v2
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
mobilevit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
mobilevitv2
Fix MobileViTV2 checkpoint name (#24018)
2023-06-05 18:12:45 +02:00
mpnet
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
mt5
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
musicgen
Add Musicgen (#24109)
2023-06-29 14:48:59 +01:00
mvp
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
nat
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
nezha
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
nllb
🚨🚨🚨 [NLLB Tokenizer] Fix the prefix tokens 🚨🚨🚨 (#22313)
2023-04-04 14:53:06 +02:00
nllb_moe
tests: Fix flaky test for NLLB-MoE (#22880)
2023-04-21 17:09:40 +01:00
nystromformer
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
oneformer
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
open_llama
add open-llama model with ckpt (#22795)
2023-04-28 11:01:32 -04:00
openai
TF: standardize test_model_common_attributes for language models (#23457)
2023-06-13 17:51:37 +01:00
opt
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
owlvit
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
pegasus
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
pegasus_x
update_pip_test_mapping (#22606)
2023-04-06 17:56:06 +02:00
perceiver
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
phobert
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
pix2struct
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
plbart
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
poolformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
prophetnet
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
qdqbert
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
rag
Big TF test cleanup (#24282)
2023-06-16 15:40:49 +01:00
realm
Unpin numba (#23162)
2023-05-31 14:59:30 +01:00
reformer
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
regnet
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
rembert
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
resnet
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
retribert
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
roberta
Clean load keys (#24505)
2023-06-27 14:45:40 -04:00
roberta_prelayernorm
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
roc_bert
Move is_pipeline_test_to_skip to specific model test classes (#21999)
2023-03-14 10:03:02 +01:00
roformer
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
rwkv
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
sam
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
segformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
sew
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
sew_d
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
speech_encoder_decoder
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
speech_to_text
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
speech_to_text_2
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
speecht5
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
splinter
Make tiny model creation + pipeline testing more robust (#22500)
2023-04-06 17:45:55 +02:00
squeezebert
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
swiftformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
swin
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
swin2sr
Skip test_multi_gpu_data_parallel_forward for some model tests (#21991)
2023-03-07 14:23:36 +01:00
swinv2
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
switch_transformers
⚠️⚠️[T5Tokenize] Fix T5 family tokenizers⚠️⚠️ (#24565)
2023-06-30 07:00:43 +02:00
t5
⚠️⚠️[T5Tokenize] Fix T5 family tokenizers⚠️⚠️ (#24565)
2023-06-30 07:00:43 +02:00
table_transformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
tapas
Skip a tapas (tokenization) test in past CI (#24378)
2023-06-20 18:35:45 +02:00
tapex
Update quality tooling for formatting (#21480)
2023-02-06 18:10:56 -05:00
time_series_transformer
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
timesformer
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
timm_backbone
Finishing tidying keys to ignore on load (#24535)
2023-06-27 21:35:15 -04:00
trajectory_transformer
Time to Say Goodbye, torch 1.7 and 1.8 (#22291)
2023-03-21 19:22:01 +01:00
transfo_xl
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
trocr
Generate: skip left-padding tests on old models (#23437)
2023-05-18 11:04:51 +01:00
tvlt
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
unispeech
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
unispeech_sat
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
upernet
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
van
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
videomae
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vilt
Removal of deprecated vision methods and specify deprecation versions (#24570)
2023-06-29 15:09:51 +01:00
vision_encoder_decoder
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vision_text_dual_encoder
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
visual_bert
Revert "Fix gradient checkpointing + fp16 autocast for most models" (#24420)
2023-06-22 16:11:27 +02:00
vit
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vit_hybrid
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vit_mae
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
vit_msn
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
wav2vec2
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
wav2vec2_conformer
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
wav2vec2_phoneme
Move test model folders (#17034)
2022-05-03 14:42:02 +02:00
wav2vec2_with_lm
Fix test_word_time_stamp_integration for Wav2Vec2ProcessorWithLMTest (#22800)
2023-04-17 12:41:55 +02:00
wavlm
Fix TypeError: Object of type int64 is not JSON serializable (#24340)
2023-06-27 12:15:49 +01:00
whisper
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
x_clip
Update some torchscript tests after #24505 (#24566)
2023-06-29 16:05:24 +02:00
xglm
Stop storing references to bound methods via tf.function (#24146)
2023-06-13 19:04:22 +01:00
xlm
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
xlm_prophetnet
Update expected values in XLMProphetNetModelIntegrationTest (#21957)
2023-03-06 09:15:44 +01:00
xlm_roberta
Better TF docstring types (#23477)
2023-05-24 13:52:52 +01:00
xlm_roberta_xl
Use real tokenizers if tiny version(s) creation has issue(s) (#22428)
2023-03-29 16:16:23 +02:00
xlnet
TensorFlow CI fixes (#24360)
2023-06-20 12:59:21 +01:00
xmod
Use real tokenizers if tiny version(s) creation has issue(s) (#22428)
2023-03-29 16:16:23 +02:00
yolos
Update old existing feature extractor references (#24552)
2023-06-29 10:17:36 +01:00
yoso
🔥Rework pipeline testing by removing PipelineTestCaseMeta 🚀 (#21516)
2023-02-28 19:40:57 +01:00
__init__.py
Move test model folders (#17034)
2022-05-03 14:42:02 +02:00