From 0d0ec1dbfba19af266ec41ab6bc1103edd1f2617 Mon Sep 17 00:00:00 2001
From: Yijun Lee <119404328+yijun-lee@users.noreply.github.com>
Date: Wed, 9 Oct 2024 09:56:30 +0900
Subject: [PATCH] =?UTF-8?q?=F0=9F=8C=90=20[i18n-KO]=20Translated=20`tokeni?=
 =?UTF-8?q?zation=5Futils.md`=20to=20Korean=20(#33813)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* docs: ko: tokenization_utils.md

* feat: nmt draft

* fix: manual edits
---
 docs/source/ko/_toctree.yml                   |  8 +++-
 docs/source/ko/internal/tokenization_utils.md | 39 +++++++++++++++++++
 2 files changed, 45 insertions(+), 2 deletions(-)
 create mode 100644 docs/source/ko/internal/tokenization_utils.md

diff --git a/docs/source/ko/_toctree.yml b/docs/source/ko/_toctree.yml
index 1e5859258b..f333e06f46 100644
--- a/docs/source/ko/_toctree.yml
+++ b/docs/source/ko/_toctree.yml
@@ -773,10 +773,14 @@
   - sections:
     - local: in_translation
       title: (번역중) Custom Layers and Utilities
+    - local: in_translation
+      title: (번역중) Utilities for pipelines
+    - local: internal/tokenization_utils
+      title: 토크나이저를 위한 유틸리티
     - local: internal/pipelines_utils
       title: 파이프라인을 위한 유틸리티
-    - local: in_translation
-      title: (번역중) Utilities for Tokenizers
+    - local: internal/tokenization_utils
+      title: 토크나이저를 위한 유틸리티
     - local: in_translation
       title: (번역중) Utilities for Trainer
     - local: in_translation
diff --git a/docs/source/ko/internal/tokenization_utils.md b/docs/source/ko/internal/tokenization_utils.md
new file mode 100644
index 0000000000..b5b6991047
--- /dev/null
+++ b/docs/source/ko/internal/tokenization_utils.md
@@ -0,0 +1,39 @@
+<!--Copyright 2020 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+
+# 토크나이저를 위한 유틸리티 [[utilities-for-tokenizers]]
+
+이 페이지는 토크나이저에서 사용되는 모든 유틸리티 함수들을 나열하며, 주로 [`PreTrainedTokenizer`]와 [`PreTrainedTokenizerFast`] 사이의 공통 메소드를 구현하는 [`~tokenization_utils_base.PreTrainedTokenizerBase`] 클래스와 [`~tokenization_utils_base.SpecialTokensMixin`]을 다룹니다.
+
+이 함수들 대부분은 라이브러리의 토크나이저 코드를 연구할 때만 유용합니다.
+
+## PreTrainedTokenizerBase [[transformers.PreTrainedTokenizerBase]]
+
+[[autodoc]] tokenization_utils_base.PreTrainedTokenizerBase
+   - __call__
+   - all
+
+## SpecialTokensMixin [[transformers.SpecialTokensMixin]]
+
+[[autodoc]] tokenization_utils_base.SpecialTokensMixin
+
+## Enums 및 namedtuples [[transformers.tokenization_utils_base.TruncationStrategy]]
+
+[[autodoc]] tokenization_utils_base.TruncationStrategy
+
+[[autodoc]] tokenization_utils_base.CharSpan
+
+[[autodoc]] tokenization_utils_base.TokenSpan