SUMIN/HuggingFace_transformer
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

Refine documentation for Tools (#23266)

Browse Source 
* refine documentation for Tools

* + one bugfix
This commit is contained in:
Sylvain Gugger
2023-05-10 11:03:53 -04:00
committed by Sylvain Gugger
parent 9d5b0e50f9
commit 9dd6209c9f
4 changed files with 45 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/source/en/_toctree.yml
View File

@@ -102,7 +102,7 @@
- local: community - local: community
title: Community resources title: Community resources
- local: custom_tools - local: custom_tools
title: Custom Tools title: Custom Tools and Prompts
- local: troubleshooting - local: troubleshooting
title: Troubleshoot title: Troubleshoot
title: Developer guides title: Developer guides

11
docs/source/en/custom_tools.mdx
View File

@@ -124,7 +124,9 @@ what the tool does and the second states what input arguments and return values
A good tool name and tool description are very important for the agent to correctly use it. Note that the only A good tool name and tool description are very important for the agent to correctly use it. Note that the only
information the agent has about the tool is its name and description, so one should make sure that both information the agent has about the tool is its name and description, so one should make sure that both
are precisely written and match the style of the existing tools in the toolbox. are precisely written and match the style of the existing tools in the toolbox. In particular make sure the description
mentions all the arguments expected by name in code-style, along with the expected type and a description of what they
are.
<Tip> <Tip>
@@ -466,7 +468,8 @@ The set of curated tools already has an `image_transformer` tool which is hereby
Overwriting existing tools can be beneficial if we want to use a custom tool exactly for the same task as an existing tool Overwriting existing tools can be beneficial if we want to use a custom tool exactly for the same task as an existing tool
because the agent is well-versed in using the specific task. Beware that the custom tool should follow the exact same API because the agent is well-versed in using the specific task. Beware that the custom tool should follow the exact same API
as the overwritten tool in this case. as the overwritten tool in this case, or you should adapt the prompt template to make sure all examples using that
tool are updated.
</Tip> </Tip>
@@ -627,14 +630,14 @@ In order to let others benefit from it and for simpler initialization, we recomm
namespace. To do so, just call `push_to_hub` on the `tool` variable: namespace. To do so, just call `push_to_hub` on the `tool` variable:
```python ```python
tool.push_to_hub("lysandre/hf-model-downloads") tool.push_to_hub("hf-model-downloads")
``` ```
You now have your code on the Hub! Let's take a look at the final step, which is to have the agent use it. You now have your code on the Hub! Let's take a look at the final step, which is to have the agent use it.
#### Having the agent use the tool #### Having the agent use the tool
We now have our tool that lives on the Hub which can be instantiated as such: We now have our tool that lives on the Hub which can be instantiated as such (change the user name for your tool):
```python ```python
from transformers import load_tool from transformers import load_tool

73
docs/source/en/transformers_agents.mdx
View File

@@ -19,7 +19,8 @@ can vary as the APIs or underlying models are prone to change.
</Tip> </Tip>
Transformers version v4.29.0, building on the concept of *tools* and *agents*. Transformers version v4.29.0, building on the concept of *tools* and *agents*. You can play with in
[this colab](https://colab.research.google.com/drive/1c7MHD-T1forUPGcC_jlwsIptOzpG3hSj).
In short, it provides a natural language API on top of transformers: we define a set of curated tools and design an In short, it provides a natural language API on top of transformers: we define a set of curated tools and design an
agent to interpret natural language and to use these tools. It is extensible by design; we curated some relevant tools, agent to interpret natural language and to use these tools. It is extensible by design; we curated some relevant tools,
@@ -60,10 +61,19 @@ agent.run(
## Quickstart ## Quickstart
Before being able to use `agent.run`, you will need to instantiate an agent, which is a large language model (LLM). Before being able to use `agent.run`, you will need to instantiate an agent, which is a large language model (LLM).
We recommend using the [bigcode/starcoder](https://huggingface.co/bigcode/starcoder) checkpoint as it works very well We provide support for openAI models as well as opensource alternatives from BigCode and OpenAssistant. The openAI
for the task at hand and is open-source, but please find other examples below. models perform better (but require you to have an openAI API key, so cannot be used for free); Hugging Face is
providing free access to endpoints for BigCode and OpenAssistant models.
Start by logging in to have access to the Inference API: To use openAI models, you instantiate an [`OpenAiAgent`]:
```py
from transformers import OpenAiAgent
agent = OpenAiAgent(model="text-davinci-003", api_key="<your_api_key>")
```
To use BigCode or OpenAssistant, start by logging in to have access to the Inference API:
```py ```py
from huggingface_hub import login from huggingface_hub import login
@@ -76,17 +86,22 @@ Then, instantiate the agent
```py ```py
from transformers import HfAgent from transformers import HfAgent
# Starcoder
agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder") agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoder")
# StarcoderBase
# agent = HfAgent("https://api-inference.huggingface.co/models/bigcode/starcoderbase")
# OpenAssistant
# agent = HfAgent(url_endpoint="https://api-inference.huggingface.co/models/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5")
``` ```
This is using the inference API that Hugging Face provides for free at the moment if you have your inference This is using the inference API that Hugging Face provides for free at the moment. If you have your own inference
endpoint for this model (or another one) you can replace the URL above with your URL endpoint. endpoint for this model (or another one) you can replace the URL above with your URL endpoint.
<Tip> <Tip>
We're showcasing StarCoder as the default in the documentation as the model is free to use and performs admirably well StarCoder and OpenAssistant are free to use and perform admirably well on simple tasks. However, the checkpoints
on simple tasks. However, the checkpoint doesn't hold up when handling more complex prompts. If you're facing such an don't hold up when handling more complex prompts. If you're facing such an issue, we recommend trying out the OpenAI
issue, we recommend trying out the OpenAI model which, while sadly not open-source, performs better at this given time. model which, while sadly not open-source, performs better at this given time.
</Tip> </Tip>
@@ -97,7 +112,7 @@ You're now good to go! Let's dive into the two APIs that you now have at your di
The single execution method is when using the [`~Agent.run`] method of the agent: The single execution method is when using the [`~Agent.run`] method of the agent:
```py ```py
agent.run("Draw me a picture of rivers and lakes") agent.run("Draw me a picture of rivers and lakes.")
``` ```
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width=200> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width=200>
@@ -107,7 +122,7 @@ can perform one or several tasks in the same instruction (though the more comple
the agent is to fail). the agent is to fail).
```py ```py
agent.chat("Draw me a picture of the sea then transform the picture to add an island.") agent.run("Draw me a picture of the sea then transform the picture to add an island")
``` ```
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/sea_and_island.png" width=200> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/sea_and_island.png" width=200>
@@ -118,15 +133,16 @@ agent.chat("Draw me a picture of the sea then transform the picture to add an is
Every [`~Agent.run`] operation is independent, so you can run it several times in a row with different tasks. Every [`~Agent.run`] operation is independent, so you can run it several times in a row with different tasks.
Note that your `agent` is just a large-language model, so small variations in your prompt might yield completely Note that your `agent` is just a large-language model, so small variations in your prompt might yield completely
different results. It's important to explain as clearly as possible the task you want to perform. different results. It's important to explain as clearly as possible the task you want to perform. We go more in-depth
on how to write good prompts [here](custom_tools#writing-good-user-inputs).
If you'd like to keep a state across executions or to pass non-text objects to the agent, you can do so by specifying If you'd like to keep a state across executions or to pass non-text objects to the agent, you can do so by specifying
variables that you would like the agent to use. For example, you could generate the first image of rivers and lakes, variables that you would like the agent to use. For example, you could generate the first image of rivers and lakes,
and ask the model to update that picture to add an island by doing the following: and ask the model to update that picture to add an island by doing the following:
```python ```python
picture = agent.run("Draw me a picture of rivers and lakes") picture = agent.run("Generate a picture of rivers and lakes.")
updated_picture = agent.chat("Take that `picture` and add an island to it", picture=picture) updated_picture = agent.run("Transform the image in `picture` to add an island to it.", picture=picture)
``` ```
<Tip> <Tip>
@@ -155,7 +171,7 @@ agent.run("Draw me a picture of the `prompt`", prompt="a capybara swimming in th
The agent also has a chat-based approach, using the [`~Agent.chat`] method: The agent also has a chat-based approach, using the [`~Agent.chat`] method:
```py ```py
agent.chat("Draw me a picture of rivers and lakes") agent.chat("Generate a picture of rivers and lakes")
``` ```
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width=200> <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/rivers_and_lakes.png" width=200>
@@ -197,6 +213,8 @@ agent.chat("Draw me a picture of rivers and lakes", remote=True)
### What's happening here? What are tools, and what are agents? ### What's happening here? What are tools, and what are agents?
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/diagram.png">
#### Agents #### Agents
The "agent" here is a large language model, and we're prompting it so that it has access to a specific set of tools. The "agent" here is a large language model, and we're prompting it so that it has access to a specific set of tools.
@@ -270,6 +288,7 @@ directly with the agent. We've added a few
- **Text downloader**: to download a text from a web URL - **Text downloader**: to download a text from a web URL
- **Text to image**: generate an image according to a prompt, leveraging stable diffusion - **Text to image**: generate an image according to a prompt, leveraging stable diffusion
- **Image transformation**: modify an image given an initial image and a prompt, leveraging instruct pix2pix stable diffusion - **Image transformation**: modify an image given an initial image and a prompt, leveraging instruct pix2pix stable diffusion
- **Text to video**: generate a small video according to a prompt, leveraging damo-vilab
The text-to-image tool we have been using since the beginning is a remote tool that lives in The text-to-image tool we have been using since the beginning is a remote tool that lives in
[*huggingface-tools/text-to-image*](https://huggingface.co/spaces/huggingface-tools/text-to-image)! We will [*huggingface-tools/text-to-image*](https://huggingface.co/spaces/huggingface-tools/text-to-image)! We will
@@ -278,32 +297,6 @@ continue releasing such tools on this and other organizations, to further superc
The agents have by default access to tools that reside on `huggingface-tools`. The agents have by default access to tools that reside on `huggingface-tools`.
We explain how to you can write and share your tools as well as leverage any custom tool that resides on the Hub in [following guide](custom_tools). We explain how to you can write and share your tools as well as leverage any custom tool that resides on the Hub in [following guide](custom_tools).
### Leveraging different agents
We showcase here how to use the [bigcode/starcoder](https://huggingface.co/bigcode/starcoder) model as an LLM, but
it isn't the only model available. We also support the OpenAssistant model and OpenAI's davinci models (3.5 and 4).
We're planning on supporting local language models in an ulterior version.
The tools defined in this implementation are agnostic to the agent used; we are showcasing the agents that work with
our prompts below, but the tools can also be used with Langchain, Minichain, or any other Agent-based library.
#### Example code for the OpenAssistant model
```py
from transformers import HfAgent
agent = HfAgent(url_endpoint="https://OpenAssistant/oasst-sft-1-pythia-12b", token="<HF_TOKEN>")
```
#### Example code for OpenAI models
```py
from transformers import OpenAiAgent
agent = OpenAiAgent(model="text-davinci-003", api_key="<API_KEY>")
```
### Code generation ### Code generation
So far we have shown how to use the agents to perform actions for you. However, the agent is only generating code So far we have shown how to use the agents to perform actions for you. However, the agent is only generating code

4
src/transformers/tools/agents.py
View File

@@ -264,7 +264,9 @@ class Agent:
""" """
prompt = self.format_prompt(task, chat_mode=True) prompt = self.format_prompt(task, chat_mode=True)
result = self.generate_one(prompt, stop=["Human:", "====="]) result = self.generate_one(prompt, stop=["Human:", "====="])
self.chat_history = prompt + result + "\n" self.chat_history = prompt + result
if not self.chat_history.endswith("\n"):
self.chat_history += "\n"
explanation, code = clean_code_for_chat(result) explanation, code = clean_code_for_chat(result)
print(f"==Explanation from the agent==\n{explanation}") print(f"==Explanation from the agent==\n{explanation}")