Topic 3: How RAG Works: Prepare the knowledge
How RAG works: Prepare the knowledge
Before RAG can find information and give it to an LLM, the knowledge first needs to be prepared for retrieval.
This is an important part of how RAG works. The system doesn’t simply search through a folder of documents every time someone asks a question. Information is organised and prepared so relevant content can be found efficiently when it is needed.
1. Start with trusted knowledge
The first step is deciding what information the RAG system can use.
For a Product Manager or Business Analyst, this could include:
- product requirements and specifications
- policies and business rules
- customer and market research
- product documentation
- user guides and support content
- knowledge bases
- project documentation
- approved internal reports
The quality of these sources matters. If the underlying information is outdated, incomplete, or incorrect, RAG can retrieve it and pass it to the LLM.
This means preparing a RAG solution is partly a knowledge management problem, not just a technology problem.
2. Break large documents into smaller parts
Business documents can be long. A requirements document might contain dozens of pages, while a policy or research report could contain hundreds.
Instead of treating the whole document as one piece of information, RAG systems usually divide it into smaller sections called chunks.
For example, a 50-page product requirements document might be broken into smaller chunks covering:
Customer problem | User requirements | Business rules | Functional requirements | Acceptance criteria
This makes it easier for the system to find the specific information that relates to a user’s question.
The way information is divided matters too. Chunks need enough context to make sense, but they should not be so large that irrelevant information gets mixed into the result.
3. Make the information searchable
Once the information has been broken into manageable pieces, the system prepares those pieces so they can be searched and retrieved.
Modern RAG systems often use embeddings for this.
An embedding is a numerical representation of a piece of content’s meaning. This lets the system find information that is similar in meaning, rather than relying only on matching the exact words used in a question.
For example, someone might ask:
“What happens if a customer’s identity check fails?”
The source document might instead use the phrase:
“Identity verification unsuccessful.”
A meaning-based search can help connect these two even though the wording is different.
The prepared information is commonly stored in a searchable system, often using a vector database or another search technology, so relevant chunks can be retrieved when a question is asked.
Why this preparation matters
Imagine an organisation has thousands of requirements, policies, research reports and product documents.
When someone asks a question, the RAG system needs to quickly identify the most relevant small pieces of that knowledge.
Good preparation helps make that possible.
If the knowledge is poorly organised, outdated or broken into unsuitable chunks, even a powerful LLM may receive the wrong context.
This is why the quality of a RAG response starts before the LLM generates anything.
What PMs and BAs should think about
You may not be responsible for building embeddings or configuring a vector database. But Product Managers and Business Analysts can play an important role in deciding:
- which knowledge sources should be included
- which sources should be trusted
- who owns and maintains the information
- how current the information needs to be
- which information should not be accessible
- what types of questions users need the system to answer
These decisions directly affect whether the RAG experience is useful to the people using it.
DSM Tip
RAG works best when the knowledge it searches is relevant, trusted and up to date.
A sophisticated retrieval system cannot compensate for poor source information. Start with the knowledge you want AI to rely on, then make sure that knowledge is accurate, well maintained and appropriate for the task.
