From 0611eab5e3ddcf155b7ac1fcc92f59aaa71ee17b Mon Sep 17 00:00:00 2001 From: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Date: Mon, 28 Sep 2020 08:31:46 -0400 Subject: [PATCH] Document RAG again (#7377) Do not merge before Monday --- docs/source/index.rst | 1 + docs/source/model_doc/rag.rst | 91 +++++++++++++++++++++++++++++++++++ docs/source/model_summary.rst | 21 ++++++++ utils/check_repo.py | 5 +- 4 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 docs/source/model_doc/rag.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index e806f0efff..1b9d882c71 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -231,6 +231,7 @@ conversion utilities for the following models: model_doc/lxmert model_doc/bertgeneration model_doc/layoutlm + model_doc/rag internal/modeling_utils internal/tokenization_utils internal/pipelines_utils diff --git a/docs/source/model_doc/rag.rst b/docs/source/model_doc/rag.rst new file mode 100644 index 0000000000..e4d401328c --- /dev/null +++ b/docs/source/model_doc/rag.rst @@ -0,0 +1,91 @@ +RAG +---------------------------------------------------- + +Overview +~~~~~~~~~~~~~~~~~~~~~ + +Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and +sequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate +outputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing +both retrieval and generation to adapt to downstream tasks. + +It is based on the paper `Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks +`__ by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir +Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. + +The abstract from the paper is the following: + +*Large pre-trained language models have been shown to store factual knowledge +in their parameters, and achieve state-of-the-art results when fine-tuned on +downstream NLP tasks. However, their ability to access and precisely manipulate +knowledge is still limited, and hence on knowledge-intensive tasks, their +performance lags behind task-specific architectures. Additionally, providing +provenance for their decisions and updating their world knowledge remain open +research problems. Pre-trained models with a differentiable access mechanism to +explicit nonparametric memory can overcome this issue, but have so far been only +investigated for extractive downstream tasks. We explore a general-purpose +fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine +pre-trained parametric and non-parametric memory for language generation. We +introduce RAG models where the parametric memory is a pre-trained seq2seq model and +the non-parametric memory is a dense vector index of Wikipedia, accessed with +a pre-trained neural retriever. We compare two RAG formulations, one which +conditions on the same retrieved passages across the whole generated sequence, the +other can use different passages per token. We fine-tune and evaluate our models +on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art +on three open domain QA tasks, outperforming parametric seq2seq models and +task-specific retrieve-and-extract architectures. For language generation tasks, we +find that RAG models generate more specific, diverse and factual language than a +state-of-the-art parametric-only seq2seq baseline.* + + + +RagConfig +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagConfig + :members: + + +RagTokenizer +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagTokenizer + :members: prepare_seq2seq_batch + + +Rag specific outputs +~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.modeling_rag.RetrievAugLMMarginOutput + :members: + +.. autoclass:: transformers.modeling_rag.RetrievAugLMOutput + :members: + + +RAGRetriever +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagRetriever + :members: + + +RagModel +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagModel + :members: forward + + +RagSequenceForGeneration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagSequenceForGeneration + :members: forward, generate + + +RagTokenForGeneration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. autoclass:: transformers.RagTokenForGeneration + :members: forward, generate diff --git a/docs/source/model_summary.rst b/docs/source/model_summary.rst index 6fc45ce516..acfaf243e9 100644 --- a/docs/source/model_summary.rst +++ b/docs/source/model_summary.rst @@ -672,6 +672,27 @@ DPR consists in three models: DPR's pipeline (not implemented yet) uses a retrieval step to find the top k contexts given a certain question, and then it calls the reader with the question and the retrieved documents to get the answer. +RAG +----------------------------------------------------------------------------------------------------------------------- + +.. raw:: html + + + Models + + + Doc + + +`Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks `_, +Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela + +Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models. +RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. +The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt to downstream tasks. + +The two models RAG-Token and RAG-Sequence are available for generation. + More technical aspects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/utils/check_repo.py b/utils/check_repo.py index ae6585be99..68a7c6a836 100644 --- a/utils/check_repo.py +++ b/utils/check_repo.py @@ -314,9 +314,8 @@ def check_repo_quality(): print("Checking all models are properly tested.") check_all_decorator_order() check_all_models_are_tested() - # Uncomment me when RAG is back - # print("Checking all models are properly documented.") - # check_all_models_are_documented() + print("Checking all models are properly documented.") + check_all_models_are_documented() if __name__ == "__main__":