Adding CTRL (squashed commit)

adding conversion script

adding first draft of modeling & tokenization

adding placeholder for test files

bunch of changes

registering the tokenizer/model/etc

tests

change link; something is very VERY wrong here

weird end-of-word thingy going on

i think the tokenization works now ; wrote the unit tests

overall structure works;load w next

the monster is alive!

works after some cleanup as well

adding emacs autosave to gitignore

currently only supporting the 48 layer one; seems to infer fine on my macbook

cleanup

fixing some documentation

fixing some documentation

tests passing?

now works on CUDA also

adding greedy?

adding greedy sampling

works well
This commit is contained in:
keskarnitish
2019-09-30 09:48:41 -07:00
parent 2dc8cb8734
commit dbed1c5d94
12 changed files with 1129 additions and 23 deletions

5
.gitignore vendored
View File

@@ -131,4 +131,7 @@ examples/runs
# data
/data
serialization_dir
serialization_dir
# emacs
*.*~