Update installation page and add contributing to the doc (#5084)

* Update installation page and add contributing to the doc

* Remove mention of symlinks
This commit is contained in:
Sylvain Gugger
2020-06-17 14:01:10 -04:00
committed by GitHub
parent 043f9f51f9
commit 204ebc25e6
6 changed files with 106 additions and 56 deletions

View File

@@ -38,6 +38,17 @@ Hugging Face showcasing the generative capabilities of several models. GPT is on
The original code can be found `here <https://github.com/openai/finetune-transformer-lm>`_.
Note:
If you want to reproduce the original tokenization process of the `OpenAI GPT` paper, you will need to install
``ftfy`` and ``SpaCy``::
pip install spacy ftfy==4.4.3
python -m spacy download en
If you don't install ``ftfy`` and ``SpaCy``, the :class:`transformers.OpenAIGPTTokenizer` will default to tokenize using
BERT's :obj:`BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most usage, don't
worry).
OpenAIGPTConfig
~~~~~~~~~~~~~~~~~~~~~