Answer Questions with Contextual Data Retrieval

Robust Question Answering with Chroma and OpenAI

This notebook guides you step-by-step through answering questions about a collection of data, using Chroma, an open-source embeddings database, along with OpenAI's text embeddings and chat completion API's.

Additionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, simple querying doesn't always create the best results!

Question Answering with LLMs

Large language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;

• Personal data like e-mails and notes
• Highly specialized data like archival or legal documents
• Newly created data like recent news stories

In order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as embeddings, alongside the documents themselves.

By embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach.

Setup and preliminaries

First we make sure the python dependencies we need are installed.

We use OpenAI's API's throughout this notebook. You can get an API key from https://beta.openai.com/account/api-keys

You can add your API key as an environment variable by executing the command export OPENAI_API_KEY=[REDACTED] in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below.

Dataset

Throughout this notebook, we use the SciFact dataset. This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus.

Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform.

Just asking the model

ChatGPT was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance.

We construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other.

We sample 50 claims from the dataset

We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way.

We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table.

We ask the model to directly assess the claims, without additional context.

Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.

Adding context

We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings.

First, we load the text corpus.

Loading the corpus into Chroma

The next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query.

We instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus.
Chroma can also be instantiated in a persisted configuration; learn more at the Chroma docs.

Next we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the embedding_function we specified earlier.

Retrieving context

Next we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance.

We create a new prompt, this time taking into account the additional context we retrieve from the corpus.

Then ask the model to evaluate the claims with the retrieved context.

Results

We see that the model performs better overall, and is now significantly better at correctly identifying false claims. Additionally, most NEE cases are also correctly identified now.

Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.

Filtering context on relevance

Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model.

If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence.

Now we assess the claims using this cleaner context.

Results

The model now assesses many fewer claims as True or False when there is not enough evidence present. However, it also is now much more cautious, tending to label most items as not enough evidence, biasing away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent.

Hypothetical Document Embeddings: Using hallucinations productively

We want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query.

Until now, we have queried the dataset using claims which are single sentence statements, while the corpus contains abstracts describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results.

We can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results.

This approach is called Hypothetical Document Embeddings (HyDE), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it.

TL;DR:

• you get much better matches when you embed whole abstracts rather than single sentences
• but claims are usually single sentences
• So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)

First, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess.

We hallucinate a document for each claim.

NB: This can take a while, about 7m for 100 claims. You can reduce the number of claims we want to assess to get results more quickly.

We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold.

We then ask the model to assess the claims, using the new context.

Results

Combining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often.

Conclusion

Equipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach.