SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Enable Gradient Accumulation fix across all models + trainer fully in forward() (#34283)

Browse Source 
* Enable grad accum fix across all models + trainer fully in forward()

* handle peft case

* Account for DDP: need to run scale tests

* Use accelerator state

* Quality

* Guard

* Experiment w/ only fairseq fix

* Fairseq only

* Revert multiply_grads fix

* Mult by grad accum to fully bring back solution

* Style

* Good to go now

* Skip fx tests for now

* Bookmark

* Working now
This commit is contained in:
Zach Mueller
2024-10-23 11:24:57 -04:00
committed by GitHub
parent 1fb575fcf0
commit d9f733625c
25 changed files with 81 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tests/models/cohere/test_modeling_cohere.py
View File

@@ -304,6 +304,10 @@ class CohereModelTest(ModelTesterMixin, GenerationTesterMixin, PipelineTesterMix
config_and_inputs[0].position_embedding_type = type
self.model_tester.create_and_check_model(*config_and_inputs)
@unittest.skip(reason="PR #34283 made changes to the forward function.")
def test_torch_fx_output_loss(self):
super().test_torch_fx_output_loss()
@require_bitsandbytes
@require_torch_sdpa
@require_torch_multi_gpu