Doc styling (#8067)
* Important files * Styling them all * Revert "Styling them all" This reverts commit 7d029395fdae8513b8281cbc2a6c239f8093503e. * Syling them for realsies * Fix syntax error * Fix benchmark_utils * More fixes * Fix modeling auto and script * Remove new line * Fixes * More fixes * Fix more files * Style * Add FSMT * More fixes * More fixes * More fixes * More fixes * Fixes * More fixes * More fixes * Last fixes * Make sphinx happy
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
RAG
|
||||
----------------------------------------------------
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and
|
||||
sequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate
|
||||
@@ -15,46 +15,40 @@ Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäs
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Large pre-trained language models have been shown to store factual knowledge
|
||||
in their parameters, and achieve state-of-the-art results when fine-tuned on
|
||||
downstream NLP tasks. However, their ability to access and precisely manipulate
|
||||
knowledge is still limited, and hence on knowledge-intensive tasks, their
|
||||
performance lags behind task-specific architectures. Additionally, providing
|
||||
provenance for their decisions and updating their world knowledge remain open
|
||||
research problems. Pre-trained models with a differentiable access mechanism to
|
||||
explicit nonparametric memory can overcome this issue, but have so far been only
|
||||
investigated for extractive downstream tasks. We explore a general-purpose
|
||||
fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine
|
||||
pre-trained parametric and non-parametric memory for language generation. We
|
||||
introduce RAG models where the parametric memory is a pre-trained seq2seq model and
|
||||
the non-parametric memory is a dense vector index of Wikipedia, accessed with
|
||||
a pre-trained neural retriever. We compare two RAG formulations, one which
|
||||
conditions on the same retrieved passages across the whole generated sequence, the
|
||||
other can use different passages per token. We fine-tune and evaluate our models
|
||||
on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art
|
||||
on three open domain QA tasks, outperforming parametric seq2seq models and
|
||||
task-specific retrieve-and-extract architectures. For language generation tasks, we
|
||||
find that RAG models generate more specific, diverse and factual language than a
|
||||
state-of-the-art parametric-only seq2seq baseline.*
|
||||
*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve
|
||||
state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely
|
||||
manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind
|
||||
task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge
|
||||
remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric
|
||||
memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a
|
||||
general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained
|
||||
parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a
|
||||
pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a
|
||||
pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages
|
||||
across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our
|
||||
models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,
|
||||
outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
|
||||
tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art
|
||||
parametric-only seq2seq baseline.*
|
||||
|
||||
|
||||
|
||||
RagConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagConfig
|
||||
:members:
|
||||
|
||||
|
||||
RagTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagTokenizer
|
||||
:members: prepare_seq2seq_batch
|
||||
|
||||
|
||||
Rag specific outputs
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.modeling_rag.RetrievAugLMMarginOutput
|
||||
:members:
|
||||
@@ -63,28 +57,28 @@ Rag specific outputs
|
||||
:members:
|
||||
|
||||
RagRetriever
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagRetriever
|
||||
:members:
|
||||
|
||||
|
||||
RagModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagModel
|
||||
:members: forward
|
||||
|
||||
|
||||
RagSequenceForGeneration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagSequenceForGeneration
|
||||
:members: forward, generate
|
||||
|
||||
|
||||
RagTokenForGeneration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagTokenForGeneration
|
||||
:members: forward, generate
|
||||
|
||||
Reference in New Issue
Block a user