Ita Zaporozhets
e48e5f1f13
Support reading tiktoken tokenizer.model file (#31656)
* use existing TikTokenConverter to read tiktoken tokenizer.model file
* del test file
* create titktoken integration file
* adding tiktoken llama test
* ALTNATIVE IMPLEMENTATION: supports llama 405B
* fix one char
* remove redundant line
* small fix
* rm unused import
* flag for converting from tiktokeng
* remove unneeded file
* ruff
* remove llamatiktokenconverter, stick to general converter
* tiktoken support v2
* update test
* remove stale changes
* udpate doc
* protect import
* use is_protobuf_available
* add templateprocessor in tiktokenconverter
* reverting templateprocessor from tiktoken support
* update test
* add require_tiktoken
* dev-ci
* trigger build
* trigger build again
* dev-ci
* [build-ci-image] tiktoken
* dev-ci
* dev-ci
* dev-ci
* dev-ci
* change tiktoken file name
* feedback review
* feedback rev
* applying feedback, removing tiktoken converters
* conform test
* adding docs for review
* add doc file for review
* add doc file for review
* add doc file for review
* support loading model without config.json file
* Revert "support loading model without config.json file"
This reverts commit 2753602e51c34cef2f184eb11f36d2ad1b02babb.
* remove dev var
* updating docs
* safely import protobuf
* fix protobuf import error
* fix protobuf import error
* trying isort to fix ruff error
* fix ruff error
* try to fix ruff again
* try to fix ruff again
* try to fix ruff again
* doc table of contents
* add fix for consistency.dockerfile torchaudio
* ruff
* applying feedback
* minor typo
* merging with push-ci-image
* clean up imports
* revert dockerfile consistency
2024-09-06 14:24:02 +02:00
..
2024-06-26 21:59:08 +01:00
2024-08-13 11:48:53 -04:00
2024-06-26 21:59:08 +01:00
2024-07-23 14:54:38 +08:00
2024-09-06 13:49:47 +02:00
2024-06-07 19:40:26 +01:00
2024-06-26 21:59:08 +01:00
2024-08-23 11:12:53 +01:00
2024-06-26 21:59:08 +01:00
2024-03-13 14:53:27 +01:00
2024-06-26 21:59:08 +01:00
2024-09-06 14:20:48 +02:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-03-13 14:53:27 +01:00
2024-07-17 08:37:43 +01:00
2024-07-16 09:32:01 -04:00
2024-08-16 11:20:41 +01:00
2024-06-19 10:18:08 +01:00
2024-08-23 11:12:53 +01:00
2024-08-23 11:12:53 +01:00
2024-08-06 11:33:05 +01:00
2024-08-27 18:50:27 +01:00
2024-08-12 20:20:17 +01:00
2024-08-06 11:33:05 +01:00
2024-05-22 06:40:15 +02:00
2024-06-26 21:59:08 +01:00
2024-08-28 10:26:00 +02:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-07-23 14:54:38 +08:00
2024-07-18 10:30:37 +05:30
2024-06-26 21:59:08 +01:00
2024-07-23 14:54:38 +08:00
2024-06-26 21:59:08 +01:00
2024-08-27 09:20:59 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2023-03-22 20:02:24 +01:00
2024-06-26 21:59:08 +01:00
2024-07-26 10:33:02 +02:00
2024-06-19 10:18:08 +01:00
2024-08-19 10:21:51 +01:00
2024-07-23 14:54:38 +08:00
2024-07-29 10:52:13 +01:00
2024-07-26 10:33:02 +02:00
2024-07-26 10:33:02 +02:00
2024-08-23 11:12:53 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-08-13 16:16:30 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-19 09:28:13 +01:00
2024-07-26 10:33:02 +02:00
2023-06-29 10:17:36 +01:00
2024-08-06 11:33:05 +01:00
2024-03-25 10:33:38 +01:00
2024-08-06 11:33:05 +01:00
2024-08-27 11:58:27 +01:00
2024-06-26 21:59:08 +01:00
2024-07-23 14:54:38 +08:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-09-05 11:55:08 +02:00
2024-06-26 21:59:08 +01:00
2024-07-26 10:33:02 +02:00
2024-08-06 11:33:05 +01:00
2024-07-10 13:46:31 +01:00
2024-06-19 10:18:08 +01:00
2024-06-26 21:59:08 +01:00
2024-05-22 06:40:15 +02:00
2024-06-26 21:59:08 +01:00
2024-08-16 11:34:13 +01:00
2024-09-06 12:56:59 +05:00
2024-08-30 18:17:25 +02:00
2024-08-06 11:33:05 +01:00
2024-07-24 17:36:32 +01:00
2024-06-26 21:59:08 +01:00
2024-05-22 06:40:15 +02:00
2024-07-23 10:11:12 +02:00
2024-06-26 21:59:08 +01:00
2024-07-24 17:36:32 +01:00
2024-03-25 10:33:38 +01:00
2024-08-27 21:27:21 +02:00
2024-08-13 11:48:53 -04:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-09-05 16:17:34 +02:00
2024-07-23 14:54:38 +08:00
2024-07-25 15:12:23 +02:00
2024-06-26 21:59:08 +01:00
2024-08-22 18:08:03 +02:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2024-09-04 16:18:06 +01:00
2024-09-04 16:18:06 +01:00
2024-08-28 09:24:06 +02:00
2024-08-14 16:22:06 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-07-22 18:24:43 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-03-25 10:33:38 +01:00
2024-09-06 14:24:02 +02:00
2024-08-13 10:14:39 +05:00
2024-09-04 17:41:51 +05:00
2024-08-15 11:44:19 +01:00
2024-09-05 14:43:20 +05:00
2024-06-26 21:59:08 +01:00
2024-07-16 09:32:01 -04:00
2024-08-05 15:19:42 +01:00
2024-07-26 10:33:02 +02:00
2024-07-16 09:32:01 -04:00
2024-08-19 16:01:35 +02:00
2024-08-16 11:20:41 +01:00
2024-08-23 11:12:53 +01:00
2024-07-22 18:24:43 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-23 11:12:53 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:18:58 +01:00
2024-06-26 21:59:08 +01:00
2024-08-05 15:19:42 +01:00
2024-07-26 10:33:02 +02:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-08-23 11:12:53 +01:00
2024-06-07 19:40:26 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-16 11:20:41 +01:00
2024-08-16 11:20:41 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 15:42:05 +02:00
2024-06-26 21:59:08 +01:00
2024-07-16 09:32:01 -04:00
2024-08-06 11:33:05 +01:00
2024-05-22 06:40:15 +02:00
2024-06-26 21:59:08 +01:00
2024-09-03 18:43:12 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-08-13 10:14:39 +05:00
2024-06-26 21:59:08 +01:00
2024-06-07 19:40:26 +01:00
2024-08-23 11:12:53 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-05-22 06:40:15 +02:00
2024-08-16 11:48:45 +01:00
2024-03-13 14:53:27 +01:00
2024-06-26 21:59:08 +01:00
2024-07-16 09:32:01 -04:00
2024-08-06 11:33:05 +01:00
2024-07-23 14:54:38 +08:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:18:58 +01:00
2024-08-08 15:47:24 +02:00
2024-08-06 11:18:58 +01:00
2024-09-06 17:19:29 +05:00
2024-06-26 21:59:08 +01:00
2024-08-16 11:20:41 +01:00
2024-06-26 21:59:08 +01:00
2024-06-19 10:18:08 +01:00
2024-06-21 01:48:10 -07:00
2024-06-19 10:18:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-04 10:09:24 +01:00
2024-05-22 06:40:15 +02:00
2024-07-23 17:07:31 +02:00
2024-08-06 11:33:05 +01:00
2024-06-20 14:15:01 +01:00
2024-06-07 19:40:26 +01:00
2024-08-05 08:40:58 +02:00
2024-07-16 09:32:01 -04:00
2024-06-26 21:59:08 +01:00
2024-06-11 15:47:38 +01:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-08-06 11:33:05 +01:00
2024-06-26 21:59:08 +01:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-05-22 06:40:15 +02:00
2024-07-26 10:33:02 +02:00
2024-08-06 11:18:58 +01:00
2024-08-06 11:18:58 +01:00
2024-06-19 10:18:08 +01:00
2024-06-07 19:40:26 +01:00
2024-06-19 10:18:08 +01:00
2024-08-06 11:33:05 +01:00
2024-06-26 18:46:48 +01:00
2024-08-06 11:48:32 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-07-22 18:24:43 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-07-22 18:24:43 +01:00
2024-06-26 21:59:08 +01:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-06-19 10:18:08 +01:00
2024-08-13 10:14:39 +05:00
2024-08-06 11:33:05 +01:00
2024-08-06 11:33:05 +01:00
2024-08-13 10:14:39 +05:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-05-22 06:40:15 +02:00
2024-08-06 11:33:05 +01:00
2024-06-07 19:40:26 +01:00
2024-06-07 19:40:26 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2024-07-08 13:49:21 +02:00
2024-08-06 11:33:05 +01:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-07-23 14:54:38 +08:00
2024-06-26 21:59:08 +01:00
2024-06-17 17:29:13 +01:00
2024-07-23 14:54:38 +08:00
2024-09-06 14:09:49 +02:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-04-02 10:27:26 +02:00
2024-08-28 10:26:00 +02:00
2024-08-28 10:26:00 +02:00
2024-06-26 21:59:08 +01:00
2024-06-04 10:09:24 +01:00
2024-06-26 21:59:08 +01:00
2024-06-26 21:59:08 +01:00
2024-08-06 11:33:05 +01:00
2022-05-03 14:42:02 +02:00