* draft changes
* update and add tests
* styling for no
* move test
* path to usable model
* update test
* small update
* update bertbased tokenizers
* don'tuse kwargs for _tokenize
* don'tuse kwargs for _tokenize
* fix copies
* update
* update test for special tokenizers
* fixup
* skip two tests
* remove pdb breakpiont()
* wowo
* rewrite custom tests
* nits
* revert chang in target keys
* fix markup lm
* update documentation of the argument
fix "UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor."
Co-authored-by: 刘长伟 <hzliuchw@corp.netease.com>
Fixing double `use_auth_token.pop` (preventing private models from
being visible).
Should fix: https://github.com/huggingface/transformers/issues/14334#issuecomment-1634527833
Repro: Have a private repo, with `vocab.json` (spread out files for the
tokenizer) and use `AutoTokenizer.from_pretrained(...,
use_auth_token="token")`.
* Add support for non-rust implemented tokenization for `__getitem__` method.
* Update for error message on adding new sub-branch for `__item__` method.
---------
Co-authored-by: liuyang17 <liuyang17@zhihu.com>
* fix for ragged list
* unpin numba
* make style
* np.object -> object
* propagate changes to tokenizer as well
* np.long -> "long"
* revert tokenization changes
* check with tokenization changes
* list/tuple logic
* catch numpy
* catch else case
* clean up
* up
* better check
* trigger ci
* Empty commit to trigger CI
* initial work
* Add other classes
* Refactor code
* Move warning and fix dynamic pipeline
* Issue warning when necessary
* Add test
* Do not skip auto tests
* Fix failing tests
* Refactor and address review comments
* Address review comments
* add draft changes
* fix failing wav2vec
* style
* make sure that the argument is saved + add tests
* style
* fixup
* update test
* default clean_up_tokenization_spaces to False for Bloom and Llama
* Update code based on review
Co-authored-by: Nicolas Patry <patry.nicolas@gmail.com>
* style
* quality
---------
Co-authored-by: Nicolas Patry <patry.nicolas@gmail.com>
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
* add warning to let the user know that the method is slower that for a fast tokenizer
* user warnings
* fix layoutlmv2
* fix layout*
* change warnings into logger.warning
* Draft new cached_file
* Initial draft for config and model
* Small fixes
* Fix first batch of tests
* Look in cache when internet is down
* Fix last tests
* Bad black, not fixing all quality errors
* Make diff less
* Implement change for TF and Flax models
* Add tokenizer and feature extractor
* For compatibility with main
* Add utils to move the cache and auto-do it at first use.
* Quality
* Deal with empty commit shas
* Deal with empty etag
* Address review comments
Fix#18385
I don't know whether `use_auth_token`, `cache_dir` and `local_files_only` should be passed to `(cls.slow_tokenizer_class)._from_pretrained`, but I guess it should.
* More informative error message
* raise dynamic error
* remove_excess_nesting application
* incorrect shape assertion for collator & function to remove excess nesting from DatasetDict
* formatting
* eliminating datasets import
* removed and relocated remove_excess_nesting to the datasets library and updated docs accordingly
* independent assert instructions
* inform user of excess nesting
