[Docs] Add language identifiers to fenced code blocks (#28955)

Add language identifiers to code blocks
This commit is contained in:
Klaus Hipp
2024-02-12 19:48:31 +01:00
committed by GitHub
parent c617f988f8
commit fe3df9d5b3
66 changed files with 137 additions and 137 deletions

View File

@@ -390,7 +390,7 @@ If your model expects those, they won't be added automatically by `apply_chat_te
text will be tokenized with `add_special_tokens=False`. This is to avoid potential conflicts between the template and
the `add_special_tokens` logic. If your model expects special tokens, make sure to add them to the template!
```
```python
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"
```

View File

@@ -310,7 +310,7 @@ Use `register_for_auto_class()` if you want the code files to be copied. If you
you don't need to call it. In cases where there's more than one auto class, you can modify the `config.json` directly using the
following structure:
```
```json
"auto_map": {
"AutoConfig": "<your-repo-name>--<config-name>",
"AutoModel": "<your-repo-name>--<config-name>",

View File

@@ -405,7 +405,7 @@ Assistant:
Therefore it is important that the examples of the custom `chat` prompt template also make use of this format.
You can overwrite the `chat` template at instantiation as follows.
```
```python
template = """ [...] """
agent = HfAgent(url_endpoint=your_endpoint, chat_prompt_template=template)

View File

@@ -72,7 +72,7 @@ pip install 'transformers[tf-cpu]'
M1 / ARM Users
You will need to install the following before installing TensorFLow 2.0
```
```bash
brew install cmake
brew install pkg-config
```

View File

@@ -41,7 +41,7 @@ You can run FastSpeech2Conformer locally with the 🤗 Transformers library.
1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers), g2p-en:
```
```bash
pip install --upgrade pip
pip install --upgrade transformers g2p-en
```

View File

@@ -50,7 +50,7 @@ this https URL.*
LayoutLMv2 depends on `detectron2`, `torchvision` and `tesseract`. Run the
following to install them:
```
```bash
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
python -m pip install torchvision tesseract
```

View File

@@ -39,7 +39,7 @@ The original code can be found [here](https://github.com/jpwang/lilt).
- To combine the Language-Independent Layout Transformer with a new RoBERTa checkpoint from the [hub](https://huggingface.co/models?search=roberta), refer to [this guide](https://github.com/jpWang/LiLT#or-generate-your-own-checkpoint-optional).
The script will result in `config.json` and `pytorch_model.bin` files being stored locally. After doing this, one can do the following (assuming you're logged in with your HuggingFace account):
```
```python
from transformers import LiltModel
model = LiltModel.from_pretrained("path_to_your_files")

View File

@@ -136,7 +136,7 @@ The same [`MusicgenProcessor`] can be used to pre-process an audio prompt that i
following example, we load an audio file using the 🤗 Datasets library, which can be pip installed through the command
below:
```
```bash
pip install --upgrade pip
pip install datasets[audio]
```

View File

@@ -54,7 +54,7 @@ The original code can be found [here](https://github.com/sweetcocoa/pop2piano).
## Usage tips
* To use Pop2Piano, you will need to install the 🤗 Transformers library, as well as the following third party modules:
```
```bash
pip install pretty-midi==0.2.9 essentia==2.1b6.dev1034 librosa scipy
```
Please note that you may need to restart your runtime after installation.

View File

@@ -64,7 +64,7 @@ Next let's have a look at one of the most important aspects when having multiple
If you use multiple GPUs the way cards are inter-connected can have a huge impact on the total training time. If the GPUs are on the same physical node, you can run:
```
```bash
nvidia-smi topo -m
```

View File

@@ -38,7 +38,7 @@ IPEX release is following PyTorch, to install via pip:
| 1.12 | 1.12.300+cpu |
Please run `pip list | grep torch` to get your `pytorch_version`, so you can get the `IPEX version_name`.
```
```bash
pip install intel_extension_for_pytorch==<version_name> -f https://developer.intel.com/ipex-whl-stable-cpu
```
You can check the latest versions in [ipex-whl-stable-cpu](https://developer.intel.com/ipex-whl-stable-cpu) if needed.

View File

@@ -39,7 +39,7 @@ Wheel files are available for the following Python versions:
| 1.12.0 | | √ | √ | √ | √ |
Please run `pip list | grep torch` to get your `pytorch_version`.
```
```bash
pip install oneccl_bind_pt=={pytorch_version} -f https://developer.intel.com/ipex-whl-stable-cpu
```
where `{pytorch_version}` should be your PyTorch version, for instance 2.1.0.
@@ -59,13 +59,13 @@ Use this standards-based MPI implementation to deliver flexible, efficient, scal
oneccl_bindings_for_pytorch is installed along with the MPI tool set. Need to source the environment before using it.
for Intel® oneCCL >= 1.12.0
```
```bash
oneccl_bindings_for_pytorch_path=$(python -c "from oneccl_bindings_for_pytorch import cwd; print(cwd)")
source $oneccl_bindings_for_pytorch_path/env/setvars.sh
```
for Intel® oneCCL whose version < 1.12.0
```
```bash
torch_ccl_path=$(python -c "import torch; import torch_ccl; import os; print(os.path.abspath(os.path.dirname(torch_ccl.__file__)))")
source $torch_ccl_path/env/setvars.sh
```
@@ -154,7 +154,7 @@ This example assumes that you have:
The snippet below is an example of a Dockerfile that uses a base image that supports distributed CPU training and then
extracts a Transformers release to the `/workspace` directory, so that the example scripts are included in the image:
```
```dockerfile
FROM intel/ai-workflows:torch-2.0.1-huggingface-multinode-py3.9
WORKDIR /workspace
@@ -286,7 +286,7 @@ set the same CPU and memory amounts for both the resource limits and requests.
After the PyTorchJob spec has been updated with values appropriate for your cluster and training job, it can be deployed
to the cluster using:
```
```bash
kubectl create -f pytorchjob.yaml
```
@@ -304,7 +304,7 @@ transformers-pytorchjob-worker-3 1/1 Running
```
The logs for worker can be viewed using `kubectl logs -n kubeflow <pod name>`. Add `-f` to stream the logs, for example:
```
```bash
kubectl logs -n kubeflow transformers-pytorchjob-worker-0 -f
```

View File

@@ -140,7 +140,7 @@ Here is the benchmarking code and outputs:
**DP**
```
```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
@@ -151,7 +151,7 @@ python examples/pytorch/language-modeling/run_clm.py \
**DDP w/ NVlink**
```
```bash
rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
@@ -162,7 +162,7 @@ torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
**DDP w/o NVlink**
```
```bash
rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \
torchrun --nproc_per_node 2 examples/pytorch/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \

View File

@@ -201,7 +201,7 @@ of 23 bits precision it has only 10 bits (same as fp16) and uses only 19 bits in
you can use the normal fp32 training and/or inference code and by enabling tf32 support you can get up to 3x throughput
improvement. All you need to do is to add the following to your code:
```
```python
import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

View File

@@ -483,7 +483,7 @@ You can also manually replicate the results of the `pipeline` if you'd like.
Now, pass your input to the model and return the `logits`:
```
```py
>>> logits = run_inference(trained_model, sample_test_video["video"])
```