Add Ernie-M Model to huggingface (#21349)

* config and tokenization(fast too) changed and ErnieEncoder added

* Slow Tokenization Added

* Tokenizer(slow) is now working and Fast Tokenizer removed

* Added Config code

* Added Base Model and utils

* ErnieMModel is now working

* All added except tests

* All tests passed except ErnieUIEM

* All tests passed

* all fixes done

* all fixes done

* fixed MAP

* fixed check_code_quality

* fixed Build PR Documentation issue

* Added changes(comments) and also updated to the latest upstream/main

* Added fixup

* Added # Copied comments

* Added fixup

* Added more comments and some nits

* Added fixup

* Fixed README_hd.md

* Added more fixes

* ErnieMTokenizer (being sentencepiece) protected and other docs edited

* Added code_quality fix

* Fixed for

* Added more fix

* modified AZ

* ernie-m tokenization test added!

* attention mask part fixed(with 0->self.config.pad_token_id)

* applied make fixup
This commit is contained in:
Susnato Dhar
2023-02-15 19:54:56 +05:30
committed by GitHub
parent 40ca13367e
commit 0c9c8472e6
30 changed files with 2343 additions and 9 deletions

View File

@@ -56,6 +56,7 @@ IGNORE_NON_TESTED = PRIVATE_MODELS.copy() + [
"Blip2QFormerModel", # Building part of bigger (tested) model.
"DetaEncoder", # Building part of bigger (tested) model.
"DetaDecoder", # Building part of bigger (tested) model.
"ErnieMForInformationExtraction",
"GraphormerEncoder", # Building part of bigger (tested) model.
"GraphormerDecoderHead", # Building part of bigger (tested) model.
"CLIPSegDecoder", # Building part of bigger (tested) model.
@@ -175,6 +176,7 @@ IGNORE_NON_AUTO_CONFIGURED = PRIVATE_MODELS.copy() + [
"Blip2ForConditionalGeneration",
"Blip2QFormerModel",
"Blip2VisionModel",
"ErnieMForInformationExtraction",
"GitVisionModel",
"GraphormerModel",
"GraphormerForGraphClassification",