Tokenizers should be framework agnostic (#8599)

* Tokenizers should be framework agnostic * Run the slow tests * Not testing * Fix documentation * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2020-11-17 14:03:03 -05:00
parent 7f3b41a306
commit 3095ee9dab
28 changed files with 73 additions and 177 deletions
--- a/src/transformers/models/rag/tokenization_rag.py
+++ b/src/transformers/models/rag/tokenization_rag.py
@@ -71,7 +71,7 @@ class RagTokenizer:
        max_length: Optional[int] = None,
        max_target_length: Optional[int] = None,
        padding: str = "longest",
-        return_tensors: str = "np",
+        return_tensors: str = None,
        truncation=True,
        **kwargs,
    ) -> BatchEncoding: