HuggingFace_transformer

Files

Loubna Ben Allal e730e12567 Update codeparrot data preprocessing (#16944 )

* add new preprocessing arguments

* add new filters

* add new filters to readme

* fix config and test count, update function names and docstrings

* reformat code

* update readme

* Update readme

* rename config_test filter

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* rename few_assignments filter

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* rename tokenizer in arguments

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* rename functions and add limit_line argument for config_test filter

* update threshold for config_test filter

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Co-authored-by: Loubna ben allal <loubnabenallal@gmail.com>

2022-05-16 14:43:25 +02:00

arguments.py

Update codeparrot data preprocessing (#16944 )

2022-05-16 14:43:25 +02:00

bpe_training.py

fix: switch from slow to generic tokenizer class (#15122 )

2022-01-12 09:12:43 -05:00

codeparrot_training.py

New features for CodeParrot training script (#16851 )

2022-04-21 18:43:46 +02:00

human_eval.py

Black preview (#17217 )