Add REALM (#13292)
* REALM initial commit * Retriever OK (Update new_gelu). * Encoder prediction score OK * Encoder pretrained model OK * Update retriever comments * Update docs, tests, and imports * Prune unused models * Make embedder as a module `RealmEmbedder` * Add RealmRetrieverOutput * Update tokenization * Pass all tests in test_modeling_realm.py * Prune RealmModel * Update docs * Add training test. * Remove completed TODO * Style & Quality * Prune `RealmModel` * Fixup * Changes: 1. Remove RealmTokenizerFast 2. Update docstrings 3. Add a method to RealmTokenizer to handle candidates tokenization. * Fix up * Style * Add tokenization tests * Update `from_pretrained` tests * Apply suggestions * Style & Quality * Copy BERT model * Fix comment to avoid docstring copying * Make RealmBertModel private * Fix bug * Style * Basic QA * Save * Complete reader logits * Add searcher * Complete searcher & reader * Move block records init to constructor * Fix training bug * Add some outputs to RealmReader * Add finetuned checkpoint variable names parsing * Fix bug * Update REALM config * Add RealmForOpenQA * Update convert_tfrecord logits * Fix bugs * Complete imports * Update docs * Update naming * Add brute-force searcher * Pass realm model tests * Style * Exclude RealmReader from common tests * Fix * Fix * convert docs * up * up * more make style * up * upload * up * Fix * Update src/transformers/__init__.py * adapt testing * change modeling code * fix test * up * up * up * correct more * make retriever work * update * make style * finish main structure * Resolve merge conflict * Make everything work * Style * Fixup * Fixup * Update training test * fix retriever * remove hardcoded path * Fix * Fix modeling test * Update model links * Initial retrieval test * Fix modeling test * Complete retrieval tests * Fix * style * Fix tests * Fix docstring example * Minor fix of retrieval test * Update license headers and docs * Apply suggestions from code review * Style * Apply suggestions from code review * Add an example to RealmEmbedder * Fix Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
This commit is contained in:
Li-Huai (Allan) Lin
committed by
GitHub
GitHub
parent
b25067d807
commit
22454ae492
@@ -240,6 +240,8 @@
|
||||
title: QDQBert
|
||||
- local: model_doc/rag
|
||||
title: RAG
|
||||
- local: model_doc/realm
|
||||
title: REALM
|
||||
- local: model_doc/reformer
|
||||
title: Reformer
|
||||
- local: model_doc/rembert
|
||||
|
||||
@@ -151,6 +151,7 @@ conversion utilities for the following models.
|
||||
1. **[PhoBERT](model_doc/phobert)** (from VinAI Research) released with the paper [PhoBERT: Pre-trained language models for Vietnamese](https://www.aclweb.org/anthology/2020.findings-emnlp.92/) by Dat Quoc Nguyen and Anh Tuan Nguyen.
|
||||
1. **[ProphetNet](model_doc/prophetnet)** (from Microsoft Research) released with the paper [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.
|
||||
1. **[QDQBert](model_doc/qdqbert)** (from NVIDIA) released with the paper [Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://arxiv.org/abs/2004.09602) by Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev and Paulius Micikevicius.
|
||||
1. **[REALM](https://huggingface.co/transformers/master/model_doc/realm.html)** (from Google Research) released with the paper [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909) by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang.
|
||||
1. **[Reformer](model_doc/reformer)** (from Google Research) released with the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.
|
||||
1. **[RemBERT](model_doc/rembert)** (from Google Research) released with the paper [Rethinking embedding coupling in pre-trained language models](https://arxiv.org/pdf/2010.12821.pdf) by Hyung Won Chung, Thibault Févry, Henry Tsai, M. Johnson, Sebastian Ruder.
|
||||
1. **[RoBERTa](model_doc/roberta)** (from Facebook), released together with the paper a [Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
|
||||
@@ -244,6 +245,7 @@ Flax), PyTorch, and/or TensorFlow.
|
||||
| ProphetNet | ✅ | ❌ | ✅ | ❌ | ❌ |
|
||||
| QDQBert | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| RAG | ✅ | ❌ | ✅ | ✅ | ❌ |
|
||||
| Realm | ✅ | ❌ | ✅ | ❌ | ❌ |
|
||||
| Reformer | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
| RemBERT | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| RetriBERT | ✅ | ✅ | ✅ | ❌ | ❌ |
|
||||
|
||||
80
docs/source/model_doc/realm.mdx
Normal file
80
docs/source/model_doc/realm.mdx
Normal file
@@ -0,0 +1,80 @@
|
||||
<!--Copyright 2022 The HuggingFace Team. All rights reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||||
the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations under the License.
|
||||
-->
|
||||
|
||||
# REALM
|
||||
|
||||
## Overview
|
||||
|
||||
The REALM model was proposed in `REALM: Retrieval-Augmented Language Model Pre-Training
|
||||
<https://arxiv.org/abs/2002.08909>`__ by Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat and Ming-Wei Chang. It's a
|
||||
retrieval-augmented language model that firstly retrieves documents from a textual knowledge corpus and then
|
||||
utilizes retrieved documents to process question answering tasks.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks
|
||||
such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network,
|
||||
requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we
|
||||
augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend
|
||||
over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the
|
||||
first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language
|
||||
modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We
|
||||
demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the
|
||||
challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both
|
||||
explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous
|
||||
methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as
|
||||
interpretability and modularity.*
|
||||
|
||||
This model was contributed by `qqaatw <https://huggingface.co/qqaatw>`__. The original code can be found `here
|
||||
<https://github.com/google-research/language/tree/master/language/realm>`__.
|
||||
|
||||
## RealmConfig
|
||||
|
||||
[[autodoc]] RealmConfig
|
||||
|
||||
## RealmTokenizer
|
||||
|
||||
[[autodoc]] RealmTokenizer
|
||||
- build_inputs_with_special_tokens
|
||||
- get_special_tokens_mask
|
||||
- create_token_type_ids_from_sequences
|
||||
- save_vocabulary
|
||||
- batch_encode_candidates
|
||||
|
||||
## RealmRetriever
|
||||
|
||||
[[autodoc]] RealmRetriever
|
||||
|
||||
## RealmEmbedder
|
||||
|
||||
[[autodoc]] RealmEmbedder
|
||||
- forward
|
||||
|
||||
## RealmScorer
|
||||
|
||||
[[autodoc]] RealmScorer
|
||||
- forward
|
||||
|
||||
## RealmKnowledgeAugEncoder
|
||||
|
||||
[[autodoc]] RealmKnowledgeAugEncoder
|
||||
- forward
|
||||
|
||||
## RealmReader
|
||||
|
||||
[[autodoc]] RealmReader
|
||||
- forward
|
||||
|
||||
## RealmForOpenQA
|
||||
|
||||
[[autodoc]] RealmForOpenQA
|
||||
- forward
|
||||
Reference in New Issue
Block a user