Using ChatGPT for Question Answering on Your Own Data

Introduction

Large language models like ChatGPT have revolutionized the field of natural language processing, offering impressive performance in a variety of tasks. However, they have their limitations, such as lacking knowledge on topics not covered in their training data and a tendency to hallucinate answers when asked about unfamiliar subjects.

Interestingly, these challenges can be addressed by combining the power of embeddings, vector search, and prompt engineering.

By adopting the techniques discussed in this article, you can effectively leverage ChatGPT to answer natural language questions on a variety of text repositories, such as your personal emails, corporate knowledge bases like Confluence, or any other collection of textual data. This allows you to create a powerful, context-aware question-answering system tailored to your specific needs, providing you with quick and accurate information retrieval from your own data sources.

Imagine being able to ask ChatGPT for the key points from an important email thread or to find a specific piece of information within your company’s documentation - all without the need to manually sift through vast amounts of data. The combination of embeddings, vector search, and prompt engineering enables you to unlock the full potential of large language models like ChatGPT and apply them to your unique use cases.

Image: Aron Brand x Midjourney v5

Understanding Embeddings for Semantic Search

Embeddings are mathematical representations of words, phrases, or even entire documents as vectors in a high-dimensional space. In simple words, as a series of numbers. The primary objective of embeddings, in the context of this article, is to encode the semantic meaning of a sequence of words in a way that sequences having a similar meaning are “close together” in this multi-dimensional space.

This property is beneficial for semantic search as it enables finding relevant results based on the meaning and context, rather than relying solely on exact keyword matches.

There are various algorithms for generating embeddings, such as Word2Vec, GloVe, and FastText, which have been widely used in natural language processing tasks. More recently, large language models like BERT, OpenAI text-embedding-ada-002, and 🤗 instructor-large have been employed to produce contextual embeddings that consider the surrounding words to better capture the meaning of a given word.

Searching for Embeddings with Vector Databases

Vector databases provide an efficient way to store and search for embeddings. These databases are designed to perform similarity searches in high-dimensional spaces, enabling the retrieval of the most semantically relevant results. By storing embeddings of documents or text passages in a vector database, a semantic search system can quickly identify the most relevant matches for a given query.

Leveraging Embeddings, Prompt Engineering and ChatGPT for Question Answering

Prompt engineering is a technique used to guide the behavior of large language models like ChatGPT by carefully crafting input prompts.

Here’s a step-by-step process of how these techniques can be combined achieve efficient question answering over an arbitrary corpus of data:

1. 1Preprocessing: Clean and preprocess the corpus of data to ensure consistency and remove irrelevant information.
2. 2Embedding Generation: Compute embeddings for each document or text passage in the corpus using a pre-trained language model like ChatGPT.
3. 3Vector Database Storage: Store the generated embeddings in a vector database, allowing for efficient similarity searches.
4. 4Query Processing: For a given user query, transform it into an embedding (in other words, a sequence of numbers) using the same pre-trained language model.
5. 5Similarity Search: Perform a similarity search in the vector database to identify the most relevant matches for the query embedding.
6. 6Prompt Engineering: Craft a prompt that combines the user’s query and the retrieved text passages, which will guide ChatGPT to generate a relevant and accurate response.
7. 7Response Generation: Feed the crafted prompt to ChatGPT and obtain the generated answer.

Coding Example with Langchain

The open-source Python package, Langchain, makes it obscenely easy to achieve the powerful combination of embeddings, vector search, and prompt engineering with large language models like ChatGPT. Langchain streamlines the process of implementing these techniques, providing a user-friendly interface and robust functionality to abstract away the complexities involved in creating a context-aware question-answering system.

To start, first install the needed packages:

%pip install langchain duckdb unstructured chromadb openai tiktoken

Now, in your Python program or Jupyter notebook, set an environment variable with your OpenAI API key:

%env OPENAI_API_KEY={your API key}

Let’s fetch some data. In this example, I’ll read it from a text file.

from langchain.document_loaders.unstructured import UnstructuredFileLoader

loader = UnstructuredFileLoader('document.txt')

documents = loader.load()

Now, split the text to chunks of 1000 characters.

from langchain.text_splitter import CharacterTextSplitter

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

The next step is to calculate embeddings for the text.

from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()

Now, let’s create a vector database and use it to index the embeddings. In this demonstration we will use a simple, in memory database that is not persistent.

from langchain.vectorstores import Chroma

db = Chroma.from_documents(texts, embeddings)

Ok, our data is indexed and we are ready for question answering! Let’s initialize the langchain chain for question answering.

from langchain.chains import VectorDBQA
from langchain.chat_models import ChatOpenAI

qa = VectorDBQA.from_chain_type(llm=ChatOpenAI(), chain_type="stuff", vectorstore=db, k=1)

Are you ready? Let’s ask questions about our data!

query = "What is the document about"
qa.run(query)

Answer: “The document appears to be about CTERA Networks, a company that provides enterprise file services for modern organizations, with a focus on cloud-driven IT solutions that prioritize security, control, and performance. The page includes information about the company’s approach, technology, products, customers, partners, resources, and team, as well as news, events, and career opportunities.”

query = "Explain CTERA's caching technology in the language of a pirate"
qa.run(query)

Answer: “Arrrr, me heartie! Listen up and I’ll tell you about CTERA’s smart edge caching technology. It’s like having a secret stash of treasure at the edge of your network, so that when ye need to access your files, they’re already close at hand. This means faster access to your data, without having to wait for it to travel long distances from the cloud. And with CTERA’s caching technology, you can also avoid running out of space on your storage ship by automatically removing old, unused data from the cache. So hoist the sails, mateys, and get ready to sail the high-speed seas with CTERA’s smart edge caching technology!”

Conclusion

In this article, I have demonstrated how to harness the capabilities of large language models like ChatGPT by combining embeddings, vector search, and prompt engineering. These techniques allow for the creation of a powerful and context-aware question-answering system that can be applied to a wide range of text repositories, including personal emails, corporate knowledge bases, and other textual data. By following the steps outlined in this article, you can extend ChatGPT’s potential to provide accurate and relevant information retrieval on your own dataset, enabling efficient and effective natural language processing applications tailored to your specific needs.