r-terada 2f53ab5745 Add sudachi and jumanpp tokenizers for bert_japanese (#19043) ...

* add sudachipy and jumanpp tokenizers for bert_japanese

* use ImportError instead of ModuleNotFoundError in SudachiTokenizer and JumanppTokenizer

* put test cases of test_tokenization_bert_japanese in one line

* add require_sudachi and require_jumanpp decorator for testing

* add sudachi and pyknp(jumanpp) to dependencies

* remove sudachi_dict_small and sudachi_dict_full from dependencies

* empty commit for ci

2022-10-05 11:41:37 -04:00

..

benchmark

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

deepspeed

deprecate is_torch_bf16_available (#17738)

2022-06-20 08:40:11 -04:00

extended

Update self-push workflow (#17177)

2022-05-13 16:28:00 +02:00

fixtures

add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629)

2022-04-11 11:06:10 +02:00

generation

Add stop sequence to text generation pipeline (#18444)

2022-09-30 14:26:51 +01:00

mixed_int8

[bnb] Minor modifications (#18631)

2022-08-17 00:48:10 +02:00

models

Add sudachi and jumanpp tokenizers for bert_japanese (#19043)

2022-10-05 11:41:37 -04:00

onnx

Add SegFormer ONNX support (#18006)

2022-08-31 20:58:44 +02:00

optimization

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

pipelines

Add stop sequence to text generation pipeline (#18444)

2022-09-30 14:26:51 +01:00

repo_utils

Add some tests for check_dummies (#19146)

2022-09-21 14:54:09 -04:00

sagemaker

transformers-cli login => huggingface-cli login (#18490)

2022-08-06 09:42:55 +02:00

tokenization

fix train_new_from_iterator in the case of byte-level tokenizers (#17549)

2022-06-08 15:30:41 +02:00

trainer

Run torchdynamo tests (#19056)

2022-09-15 11:10:16 -07:00

utils

Added tests for yaml and json parser (#19219)

2022-09-27 16:25:57 -04:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_configuration_common.py

Add tests for legacy load by url and fix bugs (#19078)

2022-09-16 23:20:02 +02:00

test_feature_extraction_common.py

Add tests for legacy load by url and fix bugs (#19078)

2022-09-16 23:20:02 +02:00

test_modeling_common.py

Poc to use safetensors (#19175)

2022-09-30 10:58:04 -04:00

test_modeling_flax_common.py

[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* (#18361)

2022-09-09 15:18:56 +02:00

test_modeling_tf_common.py

[TensorFlow] Adding GroupViT (#18020)

2022-09-29 10:48:04 +01:00

test_sequence_feature_extraction_common.py

Some tests misusing assertTrue for comparisons fix (#16771)

2022-04-19 14:44:08 +02:00

test_tokenization_common.py

More tests for regression in cached non existence (#19216)

2022-09-27 09:36:34 -04:00