Build Question Answering System with Langchain and AnalyticDB

Question Answering with Langchain, AnalyticDB and OpenAI

This notebook presents how to implement a Question Answering system with Langchain, AnalyticDB as a knowledge based and OpenAI embeddings. If you are not familiar with AnalyticDB, it’s better to check out the Getting_started_with_AnalyticDB_and_OpenAI.ipynb notebook.

This notebook presents an end-to-end process of:

• Calculating the embeddings with OpenAI API.
• Storing the embeddings in an AnalyticDB instance to build a knowledge base.
• Converting raw text query to an embedding with OpenAI API.
• Using AnalyticDB to perform the nearest neighbour search in the created collection to find some context.
• Asking LLM to find the answer in a given context.

All the steps will be simplified to calling some corresponding Langchain methods.

Prerequisites

For the purposes of this exercise we need to prepare a couple of things:
AnalyticDB cloud instance.
Langchain as a framework.
An OpenAI API key.

Install requirements

This notebook requires the following Python packages: openai, tiktoken, langchain and psycopg2cffi.

• openai provides convenient access to the OpenAI API.
• tiktoken is a fast BPE tokeniser for use with OpenAI's models.
• langchain helps us to build applications with LLM more easily.
• psycopg2cffi library is used to interact with the vector database, but any other PostgreSQL client library is also acceptable.

Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://platform.openai.com/account/api-keys ).

Once you get your key, please add it to your environment variables as OPENAI_API_KEY by running following command:

Prepare your AnalyticDB connection string

To build the AnalyticDB connection string, you need to have the following parameters: PG_HOST, PG_PORT, PG_DATABASE, PG_USER, and PG_PASSWORD. You need to export them first to set correct connect string. Then build the connection string.

Load data

In this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with AnalyticDB being the knowledge base.

Chain definition

Langchain is already integrated with AnalyticDB and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.

At this stage all the possible answers are already stored in AnalyticDB, so we can define the whole QA chain.

Search data

Once the data is put into AnalyticDB we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in AnalyticDB. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model.

Custom prompt templates

The stuff chain type in Langchain uses a specific prompt with question and context documents incorporated. This is what the default prompt looks like:

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:

We can, however, provide our prompt template and change the behaviour of the OpenAI LLM, while still using the stuff chain type. It is important to keep {context} and {question} as placeholders.

Experimenting with custom prompts

We can try using a different prompt template, so the model:

1. Responds with a single-sentence answer if it knows it.
2. Suggests a random song title if it doesn't know the answer to our question.