HuggingFace_transformer

Files

théo gigant fed27ffc7e Adding FlaxNoRepeatNGramLogitsProcessor (#29677 )

* fix issue with logit processor in beam search in Flax

* adding FlaxNoRepeatNGramLogitsProcessor class + unit test

* style correction and code verification

* add FlaxNoRepeatNGramLogitsProcessor to the test_processor_list and test_processor_list_jitted tests

* fix an issue where ngrams are banned only if they appear ==1 time + update description of get_previous_ngrams

* replace non-jit compatible masking of ngrams that are not yet generated with jittable version

* Revert "fix issue with logit processor in beam search in Flax"

This reverts commit 09b70d7e4dc32d0cc4db61af09a835a9cd238b50.

* add FlaxNoRepeatNGramLogitsProcessor to _get_logits_processor

* change the method of casting to boolean of banned tokens indices

* fix code style

* remove some useless operations + significantly faster computation of update indices using jax.lax.fori_loop

* remove useless loop iterations

* set some variables that were calculated and used multiple times

* fix format

2024-04-02 11:39:33 +02:00

__init__.py

[Test refactor 1/5] Per-folder tests reorganization (#15725 )

2022-02-23 15:46:28 -05:00

test_beam_constraints.py

Generate: move generation_*.py src files into generation/*.py (#20096 )

2022-11-09 15:34:08 +00:00

test_beam_search.py

Time to Say Goodbye, torch 1.7 and 1.8 (#22291 )

2023-03-21 19:22:01 +01:00

test_configuration_utils.py

Generate: get generation mode from the generation config instance 🧼 (#29441 )