Ita Zaporozhets e48e5f1f13 Support reading tiktoken tokenizer.model file (#31656) ...

* use existing TikTokenConverter to read tiktoken tokenizer.model file

* del test file

* create titktoken integration file

* adding tiktoken llama test

* ALTNATIVE IMPLEMENTATION: supports llama 405B

* fix one char

* remove redundant line

* small fix

* rm unused import

* flag for converting from tiktokeng

* remove unneeded file

* ruff

* remove llamatiktokenconverter, stick to general converter

* tiktoken support v2

* update test

* remove stale changes

* udpate doc

* protect import

* use is_protobuf_available

* add templateprocessor in tiktokenconverter

* reverting templateprocessor from tiktoken support

* update test

* add require_tiktoken

* dev-ci

* trigger build

* trigger build again

* dev-ci

* [build-ci-image] tiktoken

* dev-ci

* dev-ci

* dev-ci

* dev-ci

* change tiktoken file name

* feedback review

* feedback rev

* applying feedback, removing tiktoken converters

* conform test

* adding docs for review

* add doc file for review

* add doc file for review

* add doc file for review

* support loading model without config.json file

* Revert "support loading model without config.json file"

This reverts commit 2753602e51c34cef2f184eb11f36d2ad1b02babb.

* remove dev var

* updating docs

* safely import protobuf

* fix protobuf import error

* fix protobuf import error

* trying isort to fix ruff error

* fix ruff error

* try to fix ruff again

* try to fix ruff again

* try to fix ruff again

* doc table of contents

* add fix for consistency.dockerfile torchaudio

* ruff

* applying feedback

* minor typo

* merging with push-ci-image

* clean up imports

* revert dockerfile consistency

2024-09-06 14:24:02 +02:00

..

agents

Multi agents with manager (#32687)

2024-09-04 17:30:54 +02:00

benchmark

…

bettertransformer

Fixed malapropism error (#26660)

2023-10-09 11:04:57 +02:00

deepspeed

Revert PR 32299, flag users when Zero-3 was missed (#32851)

2024-08-16 12:35:41 -04:00

extended

Skip tests properly (#31308)

2024-06-26 21:59:08 +01:00

fixtures

Implementation of SuperPoint and AutoModelForKeypointDetection (#28966)

2024-03-19 14:43:02 +00:00

fsdp

🚨🚨🚨 Update min version of accelerate to 0.26.0 (#32627)

2024-08-20 11:42:36 +02:00

generation

Fix: StaticCache & inputs_embeds (#32932)

2024-09-06 12:56:59 +05:00

models

Support reading tiktoken tokenizer.model file (#31656)

2024-09-06 14:24:02 +02:00

optimization

fix: Fixed the 1st argument name in classmethods (#31907)

2024-07-11 12:11:50 +01:00

peft_integration

FIX [CI]: Fix failing tests for peft integration (#29330)

2024-02-29 03:56:16 +01:00

pipelines

Add assistant prefill for chat templates and TextGenerationPipeline (#33198)

2024-09-02 13:23:47 +01:00

quantization

Add Qwen2Moe GGUF loading support (#33264)

2024-09-05 17:42:03 +02:00

repo_utils

Refactor CI: more explicit (#30674)

2024-08-30 18:17:25 +02:00

sagemaker

Fixed log messages that are resulting in TypeError due to too many arguments (#32017)

2024-07-17 10:56:44 +01:00

tokenization

#32184 save total_vocab_size (#32240)

2024-08-05 09:22:48 +02:00

trainer

Only disallow DeepSpeed Zero-3 for auto bs finder (#31731)

2024-09-03 09:16:28 -04:00

utils

support loading model without config.json file (#32356)

2024-09-06 13:49:47 +02:00

__init__.py

…

test_backbone_common.py

Align backbone stage selection with out_indices & out_features (#27606)

2023-12-20 18:33:17 +00:00

test_configuration_common.py

Refactor: Removed un-necessary object base class (#32230)

2024-07-26 10:33:02 +02:00

test_feature_extraction_common.py

Split common test from core tests (#24284)

2023-06-15 07:30:24 -04:00

test_image_processing_common.py

Update kwargs validation for preprocess with decorator (#32024)

2024-08-06 11:33:05 +01:00

test_image_transforms.py

fix: center_crop occasionally outputs off-by-one dimension matrix (#30934)

2024-05-21 13:56:52 +01:00

test_modeling_common.py

Llava Onevision: add model (#32673)

2024-09-05 14:43:20 +05:00

test_modeling_flax_common.py

add sdpa to ViT [follow up of #29325] (#30555)

2024-05-16 10:56:11 +01:00

test_modeling_tf_common.py

Port IDEFICS to tensorflow (#26870)

2024-05-13 15:59:46 +01:00

test_pipeline_mixin.py

Config: unified logic to retrieve text config (#33219)

2024-09-04 12:03:30 +01:00

test_processing_common.py

Modify ProcessorTesterMixin for better generalization (#32637)

2024-08-13 11:48:53 -04:00

test_sequence_feature_extraction_common.py

Fix typo (#25966)

2023-09-05 10:12:25 +02:00

test_tokenization_common.py

Add assistant prefill for chat templates and TextGenerationPipeline (#33198)

2024-09-02 13:23:47 +01:00