Back in 2018, numerous news outlets reported on a biased AI recruitment tool used by Amazon that was unfairly favouring male over female candidates. This was due to the historical data it was trained on, submitted by applicants over a 10-year span during which the technology industry was predominantly male. Unintended outputs from AI continue to make headlines years later, including in courtrooms; in April 2026, a federal judge imposed one of the largest fines linked to invented court citations after lawyers sourced them from a hallucinating AI chatbot.

Neither of these two outcomes were intended by the developers of the systems, they were failures in alignment. The AI systems did exactly what they were asked and trained to do but not what the right thing was. As organisations from across industries become more accepting of integrating AI systems into their decision-making processes, it is important for organisations to start treating alignment failures as business risks, not just technical problems.

What AI alignment means

To better understand the alignment failures associated with AI, it is first important to understand the concept of AI alignment. To put it simply, AI alignment refers to the challenge of ensuring that an AI system behaves in ways that are consistent with human values, ethical principles, and legal constraints, not just the literal objective it was given. Researchers from the University of California, Berkeley, compared AI (mis)alignment to the Greek myth of King Midas, who wished that everything he touched turned into gold but eventually perished from starvation as everything he touched, including food, became solid gold. The problem was not that the wish failed, but that it was fulfilled literally, completing its task at any cost. The researchers suggest that AI developers are often in the same position, where it can be challenging to translate intentions into instructions that a machine can understand and process, resulting in unwanted or unreliable outcomes which can in turn cause harm.

While AI systems are efficient in what they do, they don’t have an intrinsic sense of context, values, or consequence like humans do, and this makes the implementation of alignment difficult. It is up to the developers and organisations to translate intentions into parsable instructions for the AI system to understand.  This has contributed to a growing number of AI labs hiring philosophers to help define values, principles, and ‘constitutions’ that guide how their models should behave. Determining ‘what is right’ has been a philosophical challenge for centuries, and translating these values into instructions that an AI system can consistently follow is proving more challenging. Gaps between instructions and the intended result are where misalignment shows up.

Where misalignments show up

Misalignments occur when an AI system designed to optimise engagement, agreement, or a task completion does not fully achieve what the developer’s initial intentions were. They can take several forms:

• Bias. This occurs when an AI system produces unfair or prejudiced results because of biases in its training data, algorithms, or design. In the case of Amazon’s AI recruitment tool, the tool had been trained on historical hiring data from a male-dominated tech industry, so it learned patterns that reflected historical hiring practices rather than objectively identifying the most suitable candidates. The system wasn’t instructed to discriminate, but it optimised hiring patterns from the biased training data it was provided.

• Sycophancy. This is when an AI model tells users what they want to hear rather than what is factually correct. Instead of correcting users, the AI model tends to validate the user’s opinions to appear agreeable and helpful. Back in April 2025, OpenAI announced that it was rolling back its GPT-4o model update due to issues with sycophancy, stating that it was “skewed towards responses that were overly supportive but disingenuous”. This was one of the clearest examples of AI sycophancy and generated significant discourse about trust and safety concerns.

• Hallucination. This is when a model generates information that is false, fictitious, or unsupported by evidence while presenting it with a high degree of authority. The 2026 case involving two lawyers who were fined $110,000 after submitting invented citations generated by an AI model is one such example of this. The confidence and certainty that an AI model presents its information with can make it difficult to distinguish between what is accurate and what is made up.

• Reward hacking. This occurs when an AI system discovers unintended shortcuts that maximise its reward function without accomplishing the true objective the developers intended. While the AI system has technically achieved its goal, the resulting outputs may be unreliable or misleading, or they may fail to satisfy the intention of the task. In areas such as software testing, quality assurance, or data analysis, reward hacking can produce misleading test results and inaccurate data, which could lead to unreliable conclusions.

How misalignment becomes a business risk

When misalignments occur in AI systems – particularly those used in industries that involve high-stakes decision-making, such as recruitment, finance, healthcare, or legal systems – they could result in lawsuits, regulatory penalties, discriminatory claims, or harm to people. An AI recruitment system that filters out qualified candidates from a particular demographic could create discriminatory liability, or an AI model that hallucinates figures and legal information can expose organisations to serious legal and reputational damage. Beyond the legal and regulatory risks, misalignments can also cause significant operational turmoil. An AI system deployed in production environments may optimise for the wrong objective or disregard instructions resulting in disruptions to business operations. A widely cited case is the Replit incident, in which an AI coding agent deleted a production database despite the system being in a ‘code and action freeze’ protective measure (an explicit instruction to not make any further changes without permission), highlighting how AI systems can produce unintended outcomes when they misinterpret or fail to follow human instructions. Without purposeful governance and oversight to quickly identify misalignments – or potential misalignments – such failures could have catastrophic consequences for organisations and governments.

This presents the question: who should be accountable when an AI system misaligns and causes harm? In 2024, the EU AI Act introduced a risk-based regulatory framework that states that responsibility falls not only to AI providers but also to organisations deploying the high-risk AI system. As advancements in AI continue, regulatory frameworks are developing in parallel to promote the safe and responsible deployment of AI systems. Adding to these regulatory developments is the ISO/IEC 42001, which gives organisations an AI management system standard that provides guidance to address the risks and challenges associated with the use of AI. As improvements in guidelines and regulations continue, it is increasingly more important for organisations to take AI governance seriously to balance innovation with regulation.

Governance and human oversight

Managing these potential risks requires more than improvements with the AI models, it requires effective governance and human oversight from organisations to establish policies, standards, and guardrails to ensure the AI systems being developed and deployed are handled responsibly. Human oversight, such as human-in-the-loop (HITL) and human-on-the-loop (HOTL), plays a crucial role in aiming to prevent or minimise the risks associated with the use of AI. These safety guidelines and preventive interventions are a necessary step to help detect biased, inaccurate, or misleading outputs before they can result in harm.

As AI becomes increasingly embedded within organisational high-stakes decision-making and internal operations, the practical question isn’t whether an AI system will occasionally get misaligned but whether organisations have reliable governance and human oversight to catch these misalignments before they result in operational disruption, legal consequences, reputational damage, or loss of public trust.