Mahdi Baghbanzadeh c61fcde910 Enhance DataCollatorForLanguageModeling with Configurable Token Replacement Probabilities (#35251) ...

* DataCollatorForLanguageModeling class was updated with new parameters that provides more control over the token masking and relacing

* DataCollatorForLanguageModeling class was updated with new parameters that provides more control over the token masking and relacing

* Addressed review comments, modified the docstring and made a test for the DataCollatorForLanguageModeling

2025-01-14 17:01:10 +00:00

..

agents

Change is_soundfile_availble to is_soundfile_available (#35030)

2025-01-03 14:37:42 +01:00

benchmark

[Test refactor 1/5] Per-folder tests reorganization (#15725)

2022-02-23 15:46:28 -05:00

bettertransformer

Fixed malapropism error (#26660)

2023-10-09 11:04:57 +02:00

deepspeed

Use inherit tempdir makers for tests + fix failing DS tests (#35600)

2025-01-10 10:01:58 -05:00

extended

[tests] skip tests for xpu (#33553)

2024-09-19 19:28:04 +01:00

fixtures

Implementation of SuperPoint and AutoModelForKeypointDetection (#28966)

2024-03-19 14:43:02 +00:00

fsdp

[tests] make cuda-only tests device-agnostic (#35607)

2025-01-13 14:48:39 +01:00

generation

[tests] make cuda-only tests device-agnostic (#35607)

2025-01-13 14:48:39 +01:00

models

Add-helium (#35669)

2025-01-13 18:41:15 +01:00

optimization

fix: Fixed the 1st argument name in classmethods (#31907)

2024-07-11 12:11:50 +01:00

peft_integration

added logic for deleting adapters once loaded (#34650)

2025-01-06 18:36:40 +00:00

pipelines

[tests] make cuda-only tests device-agnostic (#35607)

2025-01-13 14:48:39 +01:00

quantization

Fix : add require_read_token for gemma2 gated model (#35687)

2025-01-14 11:47:05 +01:00

repo_utils

Fix modular edge case + modular sorting order (#35562)

2025-01-09 17:17:52 +01:00

sagemaker

Trainer - deprecate tokenizer for processing_class (#32385)

2024-10-02 14:08:46 +01:00

tokenization

tokenizer train from iterator without pre_tokenizers (#35396)

2025-01-09 15:34:43 +01:00

tp

Simplify Tensor Parallel implementation with PyTorch TP (#34184)

2024-11-18 19:51:49 +01:00

trainer

Enhance DataCollatorForLanguageModeling with Configurable Token Replacement Probabilities (#35251)

2025-01-14 17:01:10 +00:00

utils

Enable different torch dtype in sub models (#34873)

2025-01-13 13:42:08 +01:00

__init__.py

GPU text generation: mMoved the encoded_prompt to correct device

2020-01-06 15:11:12 +01:00

test_backbone_common.py

Align backbone stage selection with out_indices & out_features (#27606)

2023-12-20 18:33:17 +00:00

test_configuration_common.py

Load sub-configs from composite configs (#34410)

2024-11-05 11:34:01 +01:00

test_feature_extraction_common.py

Split common test from core tests (#24284)

2023-06-15 07:30:24 -04:00

test_image_processing_common.py

Fix Qwen2VL processor to handle odd number of frames (#35431)

2025-01-08 13:49:00 +01:00

test_image_transforms.py

fix: center_crop occasionally outputs off-by-one dimension matrix (#30934)

2024-05-21 13:56:52 +01:00

test_modeling_common.py

[tests] make cuda-only tests device-agnostic (#35607)

2025-01-13 14:48:39 +01:00

test_modeling_flax_common.py

🚨All attention refactor🚨 (#35235)

2024-12-18 16:53:39 +01:00

test_modeling_tf_common.py

🚨All attention refactor🚨 (#35235)

2024-12-18 16:53:39 +01:00

test_pipeline_mixin.py

Add image text to text pipeline (#34170)

2024-10-31 15:48:11 -04:00

test_processing_common.py

VLMs: major clean up 🧼 (#34502)

2025-01-08 10:35:23 +01:00

test_sequence_feature_extraction_common.py

Fix typo (#25966)

2023-09-05 10:12:25 +02:00

test_tokenization_common.py

[tokenizers] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer (#35593)

2025-01-09 17:46:50 +01:00