Motoki Wu 45da7cec5a Add custom stop token ids for generation (#20727) ...

* Add StopIdStoppingCriteria

* add a working test for stop id criteria

* add to global scope

* add stop_ids to generate

* add pipeline test

* use tokenizer encode in test

* add test to generation utils

* reformat

* fixup

* make-fix-copies

* rename to stop_token_id

* use stop_tokens instead

* add to text to text generation

* make fixup

* make repo-consistency

* Add support for list of ints for eos_token_id inside generation/utils.py

* Instead of having if elses, cast the eos_token_id into a List[int]

* Add List[int] support for logits_process.py

* add List[int] for beam_search.py

* add List[int] for forced_eos_token_id

* revert stop token id stopping criteria changes

* make fixup

* fix tests

* add eos_token_id to generation/utils.py and added tests test_utils.py

* add eos_token_id type hints and fix for pad tokens

* add comments

* remove some prints and remove forced false test

* fix

* put back test_stop_sequence_stopping_criteria

* remove unused import and make fixup

* add a none check

* update docstring

* add more docstring for list ints

* make fixup

2023-01-03 15:18:24 -05:00

..

benchmark

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

deepspeed

Fix tapas scatter (#20149)

2022-11-14 01:04:26 -05:00

extended

Update self-push workflow (#17177)

2022-05-13 16:28:00 +02:00

fixtures

add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629)

2022-04-11 11:06:10 +02:00

generation

Add custom stop token ids for generation (#20727)

2023-01-03 15:18:24 -05:00

mixed_int8

Add keep_in_fp32_modules support (#20683)

2022-12-13 11:59:57 +01:00

models

Enable decoder_attention_mask in generate function (#20726)

2023-01-03 09:59:08 -05:00

onnx

Add Onnx Config for PoolFormer (#20868)

2022-12-23 01:30:57 -05:00

optimization

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

pipelines

Improve OWL-ViT postprocessing (#20980)

2023-01-03 19:25:09 +03:00

repo_utils

Repo utils test (#19696)

2022-10-18 13:47:36 -04:00

sagemaker

transformers-cli login => huggingface-cli login (#18490)

2022-08-06 09:42:55 +02:00

tokenization

fix train_new_from_iterator in the case of byte-level tokenizers (#17549)

2022-06-08 15:30:41 +02:00

trainer

fix typo output not ouput in bitsandbytes trainer test (#20839)

2022-12-20 03:16:26 -05:00

utils

Fix TF nightly tests (#20507)

2022-11-30 14:47:54 +00:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_configuration_common.py

Add WhisperModel to transformers (#19166)

2022-10-05 22:28:31 +02:00

test_feature_extraction_common.py

[Vision] .to function for ImageProcessors (#20536)

2022-12-05 19:10:54 +01:00

test_image_transforms.py

Move convert_to_rgb to image_transforms module (#20784)

2022-12-15 18:47:04 +00:00

test_modeling_common.py

[Past CI] 🔥 Leave Past CI failures in the past 🔥 (#20861)

2022-12-27 18:37:25 +01:00

test_modeling_flax_common.py

Allow flax subfolder (#19902)

2022-10-26 18:33:23 +02:00

test_modeling_tf_common.py

[Tests] Improve test_attention_outputs (#20701)

2022-12-14 14:41:40 +01:00

test_sequence_feature_extraction_common.py

Some tests misusing assertTrue for comparisons fix (#16771)

2022-04-19 14:44:08 +02:00

test_tokenization_common.py

🚨 🚨 🚨 Fix Issue 15003: SentencePiece Tokenizers Not Adding Special Tokens in convert_tokens_to_string (#15775)

2022-11-02 15:45:38 -04:00