d8f6d3790a
* update everywhere * style * pipelines * switch it everywhere in tests * switch it everywhere in docs * switch in converters everywhere * update in examples * update in model docstrings * style * warnings * style * Update configuration_utils.py * fix * Update configuration_utils.py * fixes and add first test * add pipeline tests * Update test_pipelines_common.py * add config test * Update test_modeling_common.py * add new ones * post rebase * add new * post rebase adds
2.9 KiB
2.9 KiB
ExecuTorch
ExecuTorch is a platform that enables PyTorch training and inference programs to be run on mobile and edge devices. It is powered by torch.compile and torch.export for performance and deployment.
You can use ExecuTorch with Transformers with torch.export. The [~transformers.convert_and_export_with_cache] method converts a [PreTrainedModel] into an exportable module. Under the hood, it uses torch.export to export the model, ensuring compatibility with ExecuTorch.
import torch
from transformers import LlamaForCausalLM, AutoTokenizer, GenerationConfig
from transformers.integrations.executorch import(
TorchExportableModuleWithStaticCache,
convert_and_export_with_cache
)
generation_config = GenerationConfig(
use_cache=True,
cache_implementation="static",
cache_config={
"batch_size": 1,
"max_cache_len": 20,
}
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B", pad_token="</s>", padding_side="right")
model = LlamaForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", device_map="auto", dtype=torch.bfloat16, attn_implementation="sdpa", generation_config=generation_config)
exported_program = convert_and_export_with_cache(model)
The exported PyTorch model is now ready to be used with ExecuTorch. Wrap the model with [~transformers.TorchExportableModuleWithStaticCache] to generate text.
prompts = ["Simply put, the theory of relativity states that "]
prompt_tokens = tokenizer(prompts, return_tensors="pt", padding=True).to(model.device)
prompt_token_ids = prompt_tokens["input_ids"]
generated_ids = TorchExportableModuleWithStaticCache.generate(
exported_program=exported_program, prompt_token_ids=prompt_token_ids, max_new_tokens=20,
)
generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print(generated_text)
['Simply put, the theory of relativity states that 1) the speed of light is the']