add in layer gpt2 tokenizer (#20421)

* add minimal working gpt2 tokenizer

* graph mode and output equivalence tests working

* not today tensorflow. serialization test passing!

* fix style, documentation, docstrings and all that jazz

* passing consistency checks

* move keras nlp to tf dependencies

* fix tf modeling utils and gpt2 attention to enable compiling

* fix (I hope) keras nlp dependencies

* rever changes on generation

* remove debug prints

* remove redundant tf dummy objects

* add from config, get config and max length settings to address review

* let flake ignore the error on distillation you are welcome

* test from config

* add padding test

* address sgugger review
This commit is contained in:
Pi Esposito
2022-11-29 12:02:40 -03:00
committed by GitHub
parent e8d448edcf
commit fb2b45e562
11 changed files with 297 additions and 4 deletions

View File

@@ -5,7 +5,7 @@ import gc
import os
import sys
from pathlib import Path
from typing import List
from typing import List # noqa: F401
import pytorch_lightning as pl
import torch