AI Ethics: Bias, Fairness & User Autonomy
AI can influence decisions that affect people: what they see, what they are recommended, whether an application receives attention, what price they are offered, or which opportunities are presented to them.
For Product Managers and Business Analysts, ethical AI is therefore not just a technical consideration. It is also a product design and requirements consideration.
The goal is not to eliminate every possible risk before using AI. It is to understand where AI could affect people unfairly and design appropriate safeguards into the product.
Key Takeaway: AI should work for people, not unfairly against them.
1. Where can AI bias come from?
AI systems learn and make predictions using data, examples, rules and patterns. Bias can therefore enter an AI-enabled product in several ways.
| Source | What to look for |
| Historical data | Does past data contain patterns or decisions that may already have been unfair? |
| Training examples | Are some types of users, situations or outcomes poorly represented? |
| Business rules | Could a rule unintentionally disadvantage certain customers or situations? |
| Product design | Does the way AI is introduced into the journey create different experiences for different users? |
| AI outputs | Are recommendations or decisions consistently different for particular people or groups? |
The important point is that AI does not need to be intentionally designed to discriminate for unfair outcomes to occur.
2. Fairness is about outcomes, not just intent
A team may have perfectly reasonable intentions and still create an AI-enabled experience that produces unintended consequences.
When reviewing an AI feature, don’t only ask:
“Is the AI working?”
Also ask:
- Who benefits from this decision?
- Who could be disadvantaged?
- Are similar situations producing consistently different outcomes?
- Could some users be excluded or overlooked?
- What happens when AI gets the decision wrong?
This shifts the conversation from simply testing whether the technology functions to understanding how its decisions affect people.
3. Protect user autonomy
Responsible AI should not unnecessarily remove people’s ability to understand, question or influence important decisions affecting them.
The level of control required will depend on the product and the significance of the decision.
For example, a movie recommendation and a financial decision do not carry the same consequences.
For higher-impact situations, consider whether users should be able to:
Know → Understand → Question → Choose → Escalate
Know: Is it clear that AI is involved?
Understand: Can the user understand when AI is influencing a recommendation or decision?
Question: Can the user challenge or request a review of an AI-supported outcome?
Choose: Is another option or an opt-out appropriate?
Escalate: Can the user reach a person when the situation requires human judgement?
4. What should PMs and BAs do?
You may not build or train the AI model, but you can influence the business rules, requirements, user journeys and safeguards surrounding it.
When defining an AI-enabled product or requirement, ask these five questions:
| # | Ask |
| 1 | Does the user know they’re interacting with AI? |
| 2 | Do customers understand when AI is influencing a recommendation or decision? |
| 3 | Can customers question AI decisions? |
| 4 | Can they choose another option, opt out, or speak to a person when it matters? |
| 5 | Have we checked whether AI could unfairly disadvantage certain people or groups? |
These questions should become part of discovery, requirements discussions, acceptance criteria, testing and product reviews, rather than something considered only after the AI feature has been built.
Apply It to Your Product
Think about one AI feature in a product you currently work on, or one you would like to introduce.
Ask yourself:
Who could be affected by the AI’s recommendations or decisions?
Then identify:
One potential unfair outcome → One way you could detect it → One safeguard you could introduce
For example:
Risk: Some customer applications consistently receive lower AI recommendations.
Detect: Compare outcomes across relevant customer groups and scenarios.
Safeguard: Flag unusual patterns for investigation and provide an appropriate human review route.
You don’t need to solve every AI ethics question yourself. Your responsibility is to recognise when the question needs to be asked, make it visible in the product work, and involve the right specialists when necessary.
Remember
AI should work for people, not unfairly against them.
That is the heart of ethical AI product thinking.
