Topic 4: How RAG Works: Answer the question

How RAG Works: Answer the question

How RAG Works: Answer the question

Once the knowledge is prepared and searchable, RAG can use it when someone asks a question.

This is the part of RAG that happens when the user makes a request. The system searches the available knowledge, finds information that is relevant to the question, and gives that information to the LLM as additional context.

The LLM then uses this context to generate its response.

1. The user asks a question

The process starts with a normal question.

For example, a Product Manager might ask:

“What are the main customer problems identified in our onboarding research?”

Or a Business Analyst might ask:

“What are the current requirements for customer identity verification?”

The user does not need to know where the information is stored or which document contains the answer.

2. RAG searches the knowledge

The RAG system takes the question and searches the prepared knowledge.

It looks for information relevant to the question’s meaning.

For example, the answer to a question about customer identity verification might be spread across:

  • business requirements
  • compliance policies
  • process documentation
  • acceptance criteria
  • previous project decisions

Instead of giving all these documents to the LLM, RAG searches for the most relevant parts.

3. Relevant information is retrieved

The system then selects the information that appears most useful for answering the question.

These are the chunks we discussed when preparing the knowledge.

For example, RAG might retrieve:

Chunk 1: Identity verification business rules
Chunk 2: Requirements for failed verification
Chunk 3: Relevant compliance policy
Chunk 4: Acceptance criteria for the verification journey

The quality of this retrieval step matters.

If the system finds the wrong information, misses something important or retrieves outdated content, the LLM may receive poor context.

4. The question and context go to the LLM

The retrieved information is then combined with the user’s question and provided to the LLM.

Conceptually, the LLM receives something like:

User question + Relevant information retrieved by RAG

Context for generating the answer

This is an important difference between a normal LLM interaction and a RAG-enabled interaction.

The LLM is no longer relying only on its built-in knowledge. It now has specific reference information related to the user’s question.

5. The LLM generates the answer

The LLM reads the question and the retrieved context, then uses its language capabilities to generate a response.

For example, instead of giving a generic explanation of identity verification, it can answer using the requirements and policies retrieved from your organisation’s knowledge.

This can make the response more:

  • relevant to the organisation
  • specific to the user’s question
  • grounded in available information
  • useful for the task being performed

Some RAG applications can also show citations or links back to the source documents, allowing users to see where the information came from.

What does “Grounded” mean?

You will often hear the term grounded response when discussing RAG.

A grounded response is an answer that is based on the information provided to the LLM, rather than relying only on the model’s general knowledge.

Grounding can reduce the chance of unsupported answers, but it does not guarantee that the response is correct.

The retrieved information could be incomplete, outdated or misunderstood by the model. Important outputs should still be checked against the original source.

What PMs and BAs should think about

For Product Managers and Business Analysts, the important questions are not only technical ones.

Think about:

  • What questions should users be able to ask?
  • What knowledge should the system search?
  • What sources should be trusted?
  • What happens if the system cannot find enough information?
  • Should users be able to see the source behind an answer?
  • Which answers need human verification?
  • How will you know whether the answers are actually useful?

These become important product requirements and experience decisions when designing a RAG-enabled product.

DSM Tip

Think of RAG as search first, answer second.

The LLM can only work with the context it receives. Good knowledge, good retrieval and appropriate human review all contribute to a more useful and reliable RAG experience.

Shruti DSM