théo gigant fed27ffc7e Adding FlaxNoRepeatNGramLogitsProcessor (#29677) ...

* fix issue with logit processor in beam search in Flax

* adding FlaxNoRepeatNGramLogitsProcessor class + unit test

* style correction and code verification

* add FlaxNoRepeatNGramLogitsProcessor to the test_processor_list and test_processor_list_jitted tests

* fix an issue where ngrams are banned only if they appear ==1 time + update description of get_previous_ngrams

* replace non-jit compatible masking of ngrams that are not yet generated with jittable version

* Revert "fix issue with logit processor in beam search in Flax"

This reverts commit 09b70d7e4dc32d0cc4db61af09a835a9cd238b50.

* add FlaxNoRepeatNGramLogitsProcessor to _get_logits_processor

* change the method of casting to boolean of banned tokens indices

* fix code style

* remove some useless operations + significantly faster computation of update indices using jax.lax.fori_loop

* remove useless loop iterations

* set some variables that were calculated and used multiple times

* fix format

2024-04-02 11:39:33 +02:00

..

benchmark

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

bettertransformer

Fixed malapropism error (#26660)

2023-10-09 11:04:57 +02:00

deepspeed

fix failing trainer ds tests (#29057)

2024-02-16 17:18:45 +05:30

extended

[tests] make test_trainer_log_level_replica to run on accelerators with more than 2 devices (#29609)

2024-03-13 17:44:35 +00:00

fixtures

Implementation of SuperPoint and AutoModelForKeypointDetection (#28966)

2024-03-19 14:43:02 +00:00

fsdp

Add support for FSDP+QLoRA and DeepSpeed ZeRO3+QLoRA (#29587)

2024-03-13 22:03:02 +05:30

generation

Adding FlaxNoRepeatNGramLogitsProcessor (#29677)

2024-04-02 11:39:33 +02:00

models

Fix 29807 sinusoidal positional encodings in Flaubert, Informer and XLM (#29904)

2024-04-02 10:27:26 +02:00

optimization

Make schedulers picklable by making lr_lambda fns global (#21768)

2023-03-02 12:08:43 -05:00

peft_integration

FIX [CI]: Fix failing tests for peft integration (#29330)

2024-02-29 03:56:16 +01:00

pipelines

[tests] fix the wrong output in ImageToTextPipelineTests.test_conditional_generation_llava (#29975)

2024-04-01 13:08:39 +02:00

quantization

[Quantization] Quanto quantizer (#29023)

2024-03-15 11:51:29 -04:00

repo_utils

Allow # Ignore copy (#27328)

2023-12-07 10:00:08 +01:00

sagemaker

Update all references to canonical models (#29001)

2024-02-16 08:16:58 +01:00

tokenization

Remove static pretrained maps from the library's internals (#29112)

2024-03-25 10:33:38 +01:00

tools

Add support for for loops in python interpreter (#24429)

2023-06-26 09:58:14 -04:00

trainer

Rework tests to compare trainer checkpoint args (#29883)

2024-03-30 22:19:17 -04:00

utils

Remove static pretrained maps from the library's internals (#29112)

2024-03-25 10:33:38 +01:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_backbone_common.py

Align backbone stage selection with out_indices & out_features (#27606)

2023-12-20 18:33:17 +00:00

test_cache_utils.py

Generate: add tests for caches with pad_to_multiple_of (#29462)

2024-03-06 10:57:04 +00:00

test_configuration_common.py

[ PretrainedConfig] Improve messaging (#27438)

2023-11-15 14:10:39 +01:00

test_configuration_utils.py

[tests] remove deprecated tests for model loading (#29450)

2024-03-15 14:18:41 +00:00

test_feature_extraction_common.py

Split common test from core tests (#24284)

2023-06-15 07:30:24 -04:00

test_feature_extraction_utils.py

[tests] remove deprecated tests for model loading (#29450)

2024-03-15 14:18:41 +00:00

test_image_processing_common.py

Raise unused kwargs image processor (#29063)

2024-02-20 16:20:20 +01:00

test_image_processing_utils.py

[tests] remove deprecated tests for model loading (#29450)

2024-03-15 14:18:41 +00:00

test_image_transforms.py

Normalize floating point cast (#27249)

2023-11-10 15:35:27 +00:00

test_modeling_common.py

Fix FA2 tests (#29909)

2024-04-01 07:51:00 +00:00

test_modeling_flax_common.py

[Flax] Update no init test for Flax v0.7.1 (#28735)

2024-01-26 18:20:39 +00:00

test_modeling_flax_utils.py

Enable safetensors conversion from PyTorch to other frameworks without the torch requirement (#27599)

2024-01-23 10:28:23 +01:00

test_modeling_tf_common.py

Add tf_keras imports to prepare for Keras 3 (#28588)

2024-01-30 17:26:36 +00:00

test_modeling_tf_utils.py

Cast bfloat16 to float32 for Numpy conversions (#29755)

2024-03-21 14:04:11 +00:00

test_modeling_utils.py

Reimplement "Automatic safetensors conversion when lacking these files" (#29846)

2024-03-27 08:58:08 +01:00

test_pipeline_mixin.py

Image Feature Extraction pipeline (#28216)

2024-02-05 14:50:07 +00:00

test_processing_common.py

Don't save processor_config.json if a processor has no extra attribute (#28584)

2024-01-19 09:59:14 +00:00

test_sequence_feature_extraction_common.py

Fix typo (#25966)

2023-09-05 10:12:25 +02:00

test_tokenization_common.py

Remove static pretrained maps from the library's internals (#29112)

2024-03-25 10:33:38 +01:00

test_tokenization_utils.py

[tests] remove deprecated tests for model loading (#29450)

2024-03-15 14:18:41 +00:00