Topic 4: Understanding LLM limitations
Understanding LLM limitations – context, knowledge & hallucinations
Large Language Models can produce remarkably useful answers, but they also have important limitations. For Product Managers and Business Analysts, understanding these limitations is essential because AI output may influence requirements, research, prioritisation, product decisions and stakeholder communication.
The most important thing to remember is simple: an answer that sounds convincing is not necessarily correct.
LLMs only know the context available to them
An LLM does not automatically understand your organisation or project in the way you do.
Unless you provide the relevant information, it may not know your:
- product and customers
- business objectives
- requirements and constraints
- stakeholder expectations
- previous decisions
- organisational terminology
- project history
When context is missing, the LLM may fill the gaps using general patterns. The answer might sound reasonable but be completely unsuitable for your situation.
Context also has limits
LLMs can process a large amount of information, but there is still a limit to how much information a model can work with at one time. This is commonly called its context window.
Even within that context, giving an LLM a huge amount of information does not guarantee that every detail will receive equal attention.
For example, if you provide a long requirements document, workshop transcript and several research reports, don’t assume the LLM has understood every important detail correctly.
For critical work, break information into manageable parts and check the output as you go.
Knowledge may be incomplete or outdated
An LLM’s built-in knowledge is not the same as having access to a continuously updated database of facts.
Depending on the AI tool and how you are using it, the model may not know about:
- recent events
- new regulations
- current market information
- changes within your organisation
- the latest product or competitor information
- specialist or proprietary knowledge
Some AI tools can search the web, access connected company information or retrieve information from other sources. That can improve the answer, but you should still understand where the information came from and whether the source is trustworthy.
What is an AI hallucination?
A hallucination happens when an AI model generates information that appears plausible but is incorrect or made up.
For PMs and BAs, this could mean AI inventing:
- a customer insight that wasn’t present in the research
- a requirement that wasn’t agreed
- a statistic or market fact
- a source or reference
- a product capability
- an assumption presented as a fact
This can be particularly dangerous because hallucinated information may be written clearly and confidently.
Confidence is not evidence
LLMs generate responses based on patterns and probabilities. They are designed to produce useful, coherent responses, not to guarantee that every statement is true.
That means a beautifully structured answer with detailed reasoning can still contain mistakes.
So avoid using how confident the AI sounds as your measure of whether something is correct.
Instead ask: What evidence supports this?
Check what matters
You don’t need to verify every sentence an LLM generates with the same level of effort. Focus your attention on information that could affect an important decision or deliverable.
For product and BA work, pay particular attention to:
- facts and statistics
- customer evidence
- requirements
- assumptions
- calculations
- external sources
- regulatory or policy information
- recommendations that influence important decisions
Compare AI-generated output with your original research, project documents, data and trusted sources.
Keep humans responsible for decisions
LLMs are excellent tools for exploring ideas, analysing information, drafting content, and challenging your thinking.
But the LLM should not become the decision-maker.
A Product Manager still owns product judgement. A Business Analyst still needs to validate requirements and assumptions. Teams still need to understand the evidence behind important decisions.
DSM Tip
Useful does not always mean correct.
The more important the decision, requirement or recommendation, the more carefully you should validate the LLM’s output before using it.
