In this knowledge article, I will handhold and walk you through the RAG framework/pipeline for creating customized LLM with the latest knowledge base.
Synopsis:
- Introduction to RAG
- Why RAG?
- RAG Architecture
- End-2-End RAG Pipeline
- Benefits of RAG
- Pitfall
- RAG use-cases
- Conclusion
Introduction to RAG:
As we all know and use opensource LLMs and paid ones for our customised tasks with already pre-trained models up to a particular knowledge or data, and if you ask for any domain-specific or personalised answer for your specific need the LLMs won't be that efficient for your query.
Don't worry it's just a speed bumper for your driving lessons, will get you through!
RAG: Retrieval Augmented Generation is a framework or a pipeline that allows your LLM to connect with current/real-world domain-specific data. So, RAG is an architecture that allows the LLM to connect with external sources of data.
Why RAG?
- LLMs are pre-trained models with limited knowledge and are not relevant to the current date.
- Transparency can occur which can lead to misleading information.
- It can Hallucinate
So, now we got to know what is RAG, let me define the architecture of RAG.
RAG Architecture:
RAG is simply defined as retrieval augmented generation
- Retrieval
- Augmented
- Generation
Retrieval is defined as, once the user asks the query, the query should go and fetch the answers on the database this is the retrieval process.
Augmented is defined as collecting/enhancing all the answers that have been fetched from the database.
Generation is defined as once after collecting all the data from the database we pass the answer along with the prompt to the LLM, and the LLM will generate an answer based on the prompt and question given to it
There are three components involved in RAG architecture
- Ingestion
- Retrieval
- Generation
Ingestion
Ingestion or data ingestion is simply loading the data. Once the data is loaded we have to split the data. after splitting the data we have to embed the data and if required we can also construct indexing. Once this process is done we have to store this vectorised data in the vector DB, a vector DB is a database that is similar to other databases but specialised for storing embedded data.
So, Ingestion is the combination of loading, splitting, embedding, and storing the data in the DB
Ingestion => Load + Split + Embed + Storing in DB
Why is data splitting required?
if you look through the documentation of any LLM models there will be a term called Context Window for each of the LLM models, so if you load a document for a domain-specific use case that is relevant to your particular use case, to enhance the LLM performance to the current scenario. the document can be in a different context size which may exceed the context window size of the LLM models. So, to effectively bring the large document into a smaller context window fitting the context window of the LLM model we are splitting the large document into smaller chunks.
Why embedding required?
After splitting the data into smaller chunks we have to embed the chunks. As we all probably know machines understand only numeric representations of any data and not text data. So to convert the text data that we are loading into the LLM we are using a text embedding model which can be either OpenAI embeddings or open source embeddings.
Why should we store it in the vector database?
so we have converted the whole text representation of the document into a numeric representation and this numeric representation should be stored in a database for accessing the data for future purposes. let’s consider a user is asking a query to a user domain-specific question, the LLM has to go through the document and then has to generate a response according to the question asked by the user. To access the data that we have loaded already, we store the data in a vector database for efficient and accurate retrieval options.
Multiple cloud and in-memory databases are available that you can skim through the internet according to your requirements.
Retrieval
As discussed earlier, ingestion involves Load + Split + Embed + Storing in DB, after the 1st part is done here comes the process of retrieving the data from the database, based on the user query. Retrieval is a process that involves fetching the data from the vector database based on the questions asked, there are multiple techniques involved in advanced RAG for faster, accurate information retrieval.
Generation
Once the data is retrieved from the database then the data that has been retrieved will be finally passed to the LLM along with the prompt given to generate the tailored response according to the question asked by the user.
Benefits of using RAG
- Connection to the external sources of data
- More relevancy and accuracy
- Open-domain specific models
- Reduced bais or e hallucinations
- RAG doesn't require any model training
Pitfall of RAG:
The performance of the RAG is heavily dependent on the architecture itself and its knowledge base. If no proper optimization is performed in the architecture it may lead to poor performance.
RAG use-cases
- Document Q/A
- Conversational agents
- Real-time event commentary
- Content generation
- Personalised recommendation
- Virtual assistance
Conclusion
The RAG (Retrieval-Augmented Generation) application stands as a groundbreaking solution in the realm of AI-driven tools, merging the best of both retrieval and generative models to deliver highly accurate and contextually relevant responses. By leveraging a powerful combination of information retrieval and sophisticated language generation, RAG ensures that users receive precise, well-informed answers to their queries.
This innovative approach not only enhances the quality and reliability of information provided but also significantly improves user experience by minimizing response times and increasing the depth of knowledge available at their fingertips. The RAG application showcases the immense potential of AI in transforming how we access and utilize information, setting a new standard for intelligent, efficient, and user-centric AI solutions.
As we continue to refine and expand the capabilities of RAG, we anticipate even greater advancements in various fields, from customer support and content creation to research and education. The future of AI-powered applications is indeed promising, and RAG is at the forefront of this exciting evolution.
