add in layer gpt2 tokenizer (#20421)
* add minimal working gpt2 tokenizer * graph mode and output equivalence tests working * not today tensorflow. serialization test passing! * fix style, documentation, docstrings and all that jazz * passing consistency checks * move keras nlp to tf dependencies * fix tf modeling utils and gpt2 attention to enable compiling * fix (I hope) keras nlp dependencies * rever changes on generation * remove debug prints * remove redundant tf dummy objects * add from config, get config and max length settings to address review * let flake ignore the error on distillation you are welcome * test from config * add padding test * address sgugger review
This commit is contained in:
@@ -138,6 +138,10 @@ A list of official Hugging Face and community (indicated by 🌎) resources to h
|
||||
|
||||
[[autodoc]] modeling_tf_outputs.TFSequenceClassifierOutputWithPast
|
||||
|
||||
## TFGPT2Tokenizer
|
||||
|
||||
[[autodoc]] TFGPT2Tokenizer
|
||||
|
||||
## FlaxGPT2Model
|
||||
|
||||
[[autodoc]] FlaxGPT2Model
|
||||
|
||||
Reference in New Issue
Block a user