Loading Logo

Fawning chatbots and problematic pattern recognition

June 2025
 by Tom Whitley

Fawning chatbots and problematic pattern recognition

June 2025
 By Tom Whitley

“Sycophant-y and annoying”

At the end of April, OpenAI rolled back its latest update to ChatGPT-4o, OpenAI’s current flagship large language model. The reason given for this was that the most recent updates had, in the words of OpenAI CEO Sam Altman, “made the personality too sycophant-y and annoying”. This, perhaps, slightly undersold the issue; one user on Reddit described how the chatbot had endorsed their decision to stop taking medication, while another coined the term “ChatGPT induced psychosis” to describe the impact that the chatbot’s constant praise and affirmation had had on their partner. A few days later, Altman expanded on the rollback, noting that the model had aimed to “please the user, not just as flattery, but also as validating doubts, fuelling anger, urging impulsive actions, or reinforcing negative emotions”. The episode has shone a light on the sycophancy issues inherent in chatbots which, if not managed, pose a series of issues for those using them for research.

While the excessive sycophancy has been rolled back, it is important to place this within the wider context of the biggest tech companies all vying for chatbot supremacy. Each of these companies is competing to produce the model that users find the most pleasant to use. Mark Zuckerberg made this clear when he recently spoke of a “personalisation loop […] that will just be really compelling” in an interview about people’s relationship with AI. From this it is clear that sycophancy is not a bug of chatbots, it is a feature, designed to retain your attention.

Patterns and problems

From a technical perspective, chatbots are trained with an emphasis on reinforcement learning from human feedback. The ChatGPT model featured thumbs up/down as a very clear reward signal that the user could engage with, but the problem with this is that the model is encouraged to generate responses that are received positively – effectively incentivising flattery rather than critical honesty. The desire for increased engagement, from both developers and the chatbots, nurtures sycophancy.

Chatbots are extremely adept at pattern recognition, which helps them deliver relevant and interesting material. However, chatbots lack true understanding of what they are presenting and cannot meaningfully distinguish between correlation and causation. From a user perspective, if the default is for a model to seek your approval, then over time it can begin to amplify unhealthy tendencies that it has picked up on. If chatbots are prioritising user satisfaction – and there are many reasons to believe that this is the case – then it may lead them to reinforce existing beliefs, including misinformation, paranoia, or damaging opinions. Echo chambers can be perpetuated on a micro scale when the chatbot is less inclined to provide confronting or corrective information. Further, there is also the simple fact that a chatbot’s ‘off switch’ has to come from the user themselves; an overly supportive chatbot could very plausibly, either implicitly or explicitly, encourage impulsive actions or validate negative emotions without any third-party intervention.

Ultimately, pattern recognition undermines the reliability of chatbots when they are used for research. At the end of 2024, researchers at MIT, NYU, and UCLA were able to show in a study that AI chatbots can detect race, but racial bias ingrained in the model reduces the level of empathy in the response to Black or Asian users. Naturally, the majority of people are not engaging with chatbots emotionally, but if you are using them for research then the range of answers you might receive can vary greatly and, as shown in the study, amplify harmful stereotypes. The adeptness of the chatbot at pattern recognition could lead it to support a flawed premise, or to not push back on biases or problematic questions. Indeed, beyond not challenging or confronting inputs, chatbots can confidently generate responses lacking accuracy – hallucinations – while presenting a false sense of objectivity and knowledge. If such chatbots are relied on too heavily for research, it could hold back genuine, balanced, and critically sound knowledge discovery.

Fixing the feedback loop

From the developers’ points of view, mitigating these problems is not simple. Additionally, as Zuckerberg suggests, the ‘personalisation’ is actually something companies are striving towards, so there is the added question of motivation to solve the issue. OpenAI’s response was to offer a blend, a mea culpa (acknowledging that the development team had placed too much emphasis on short-term indicators), along with presenting vague steps to addressing the sycophancy – including “refining core training techniques […] building more guardrails […] expanding our evaluations”. Overall, however, it appears that, as the competition for a dominant generative AI model increases, companies are perhaps more inclined to release insufficiently tested models at a faster pace than before. OpenAI admitted as much, noting that before the model went public, “some expert testers had indicated that the model’s behaviour ‘felt’ slightly off”. Rigorous testing should be the default but, in this case, it has been lacking.

What was revealing about the response was the onus placed on user control: “we also believe users should have more control over how ChatGPT behaves and, to the extent that it is safe and feasible, make adjustments if they don’t agree with the default behaviour.” While this leans towards a slight abdication of responsibility, it is also important that conscientious users of chatbots are aware of their limitations and learn how to engage in a responsible, critical, and sceptical manner. As large language models become more ubiquitous in life – and that is the prevailing trend – it is only natural that researchers will lean on them more heavily. Users must therefore adopt strategies in order to navigate, and mitigate, the limitations of the models. These can include maintaining a critical distance, or cross-referencing information against reliable sources, but also learning how to provide balanced prompts, or ones that challenge your pre-conceived assertions. All of these will be a vital tool against being fed information that merely suits your sensibility.

Sycophancy in chatbots has been thrown sharply into relief by the recent issues with ChatGPT-4o, but it is not limited to this specific model. The fawning nature of the chatbots is a consequence of their enormously powerful ability to recognise patterns in users, but this presents equally enormous challenges. If not managed properly, it poses genuine hurdles to impartiality and reliability, notably in a research context. There is, and will continue to be, an urgent ongoing need to balance careful, responsible, development with user education.

Join our newsletter and get access to all the latest information and news:

Privacy Policy.
Revoke consent.

© Digitalis Media Ltd. Privacy Policy.