Arthur
30b3c46ff5
[split_special_tokens] Add support for split_special_tokens argument to encode ( #25081 )
...
* draft changes
* update and add tests
* styling for no
* move test
* path to usable model
* update test
* small update
* update bertbased tokenizers
* don'tuse kwargs for _tokenize
* don'tuse kwargs for _tokenize
* fix copies
* update
* update test for special tokenizers
* fixup
* skip two tests
* remove pdb breakpiont()
* wowo
* rewrite custom tests
* nits
* revert chang in target keys
* fix markup lm
* update documentation of the argument
2023-08-18 13:26:27 +02:00
..
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-06 14:31:14 -04:00
2023-06-26 18:36:47 +02:00
2023-08-09 18:28:02 +02:00
2023-08-02 20:22:36 +02:00
2023-02-06 18:10:56 -05:00
2023-02-06 18:10:56 -05:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-02-06 18:10:56 -05:00
2023-02-06 18:10:56 -05:00
2023-06-22 16:11:27 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-29 10:17:36 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-08-18 12:40:40 +02:00
2023-08-11 11:30:18 +01:00
2023-02-06 18:10:56 -05:00
2023-05-24 13:52:52 +01:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-08 10:48:45 +02:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-05-16 23:35:11 +02:00
2023-03-22 20:02:24 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-07-24 10:08:47 -04:00
2023-08-08 10:48:45 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-02-28 19:40:57 +01:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-06-29 10:17:36 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-29 10:17:36 +01:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-08-14 17:08:47 +01:00
2023-08-02 20:22:36 +02:00
2023-07-24 10:08:47 -04:00
2023-06-06 18:30:51 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-08 10:48:45 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-05-18 17:29:04 +02:00
2023-04-06 17:56:06 +02:00
2023-06-16 15:40:49 +01:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2022-12-12 13:12:13 -05:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-07-24 10:08:47 -04:00
2023-08-02 20:22:36 +02:00
2023-02-06 18:10:56 -05:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-06-22 16:11:27 +02:00
2023-08-03 11:01:10 +01:00
2023-08-18 12:40:40 +02:00
2023-08-02 20:22:36 +02:00
2023-08-18 13:26:27 +02:00
2023-08-18 13:26:27 +02:00
2023-08-18 13:26:27 +02:00
2023-06-30 16:30:33 +01:00
2023-08-11 11:30:18 +01:00
2023-06-22 16:11:27 +02:00
2023-08-17 17:08:05 +02:00
2023-08-07 17:45:41 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-16 15:40:49 +01:00
2023-04-06 17:56:06 +02:00
2023-08-16 11:49:29 +01:00
2023-08-18 13:26:27 +02:00
2023-08-11 11:30:18 +01:00
2023-08-16 18:11:24 +02:00
2023-08-02 20:22:36 +02:00
2022-07-29 08:09:09 -04:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2022-05-03 14:42:02 +02:00
2023-08-11 11:30:18 +01:00
2022-05-12 16:25:55 -04:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-08-11 11:30:18 +01:00
2023-07-24 10:08:47 -04:00
2023-08-02 20:22:36 +02:00
2023-08-17 09:06:26 +02:00
2023-08-02 20:22:36 +02:00
2023-05-24 13:52:52 +01:00
2023-07-28 18:50:15 +01:00
2023-08-02 20:22:36 +02:00
2023-06-29 10:17:36 +01:00
2023-08-02 20:22:36 +02:00
2023-04-04 14:53:06 +02:00
2023-08-17 17:21:56 +02:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-18 12:40:40 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-07-25 07:56:04 -04:00
2023-02-06 18:10:56 -05:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-06-16 15:40:49 +01:00
2023-08-02 20:22:36 +02:00
2023-05-18 11:04:51 +01:00
2023-06-29 10:17:36 +01:00
2023-08-02 20:22:36 +02:00
2023-06-29 10:17:36 +01:00
2023-08-08 10:48:45 +02:00
2023-08-08 10:48:45 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-06-22 16:11:27 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-02-06 18:10:56 -05:00
2023-06-29 16:05:24 +02:00
2023-08-02 20:22:36 +02:00
2023-08-03 14:12:07 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-07-25 07:56:04 -04:00
2023-06-29 10:17:36 +01:00
2023-08-16 17:45:02 +01:00
2023-06-29 10:17:36 +01:00
2023-08-11 13:16:01 +01:00
2023-08-17 17:08:05 +02:00
2023-07-25 07:56:04 -04:00
2023-08-02 20:22:36 +02:00
2023-06-22 16:11:27 +02:00
2023-08-02 20:22:36 +02:00
2023-07-25 07:56:04 -04:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-07-25 07:56:04 -04:00
2023-08-16 17:45:02 +01:00
2023-08-11 11:30:18 +01:00
2023-06-29 10:17:36 +01:00
2023-07-05 13:44:30 +02:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-16 17:02:57 +02:00
2023-08-02 20:22:36 +02:00
2023-08-02 20:22:36 +02:00
2023-08-16 17:45:02 +01:00
2023-08-11 15:26:45 +01:00
2023-08-02 20:22:36 +02:00
2022-05-03 14:42:02 +02:00
2023-04-17 12:41:55 +02:00
2023-08-02 20:22:36 +02:00
2023-08-17 15:30:46 +02:00
2023-08-02 20:22:36 +02:00
2023-08-18 12:40:40 +02:00
2023-08-02 20:22:36 +02:00
2023-03-06 09:15:44 +01:00
2023-05-24 13:52:52 +01:00
2023-08-02 20:22:36 +02:00
2023-08-17 18:56:34 +02:00
2023-08-02 20:22:36 +02:00
2023-08-11 11:30:18 +01:00
2023-08-02 20:22:36 +02:00
2022-05-03 14:42:02 +02:00