“Sycophant-y and annoying”
At the end of April, OpenAI rolled back its latest update to ChatGPT-4o, OpenAI’s current flagship large language model. The reason given for this was that the most recent updates had, in the words of OpenAI CEO Sam Altman, “made the personality too sycophant-y and annoying”. This, perhaps, slightly undersold the issue; one user on Reddit described how the chatbot had endorsed their decision to stop taking medication, while another coined the term “ChatGPT induced psychosis” to describe the impact that the chatbot’s constant praise and affirmation had had on their partner. A few days later, Altman expanded on the rollback, noting that the model had aimed to “please the user, not just as flattery, but also as validating doubts, fuelling anger, urging impulsive actions, or reinforcing negative emotions”. The episode has shone a light on the sycophancy issues inherent in chatbots which, if not managed, pose a series of issues for those using them for research.
While the excessive sycophancy has been rolled back, it is important to place this within the wider context of the biggest tech companies all vying for chatbot supremacy. Each of these companies is competing to produce the model that users find the most pleasant to use. Mark Zuckerberg made this clear when he recently spoke of a “personalisation loop […] that will just be really compelling” in an interview about people’s relationship with AI. From this it is clear that sycophancy is not a bug of chatbots, it is a feature, designed to retain your attention.
Patterns and problems
From a technical perspective, chatbots are trained with an emphasis on reinforcement learning from human feedback. The ChatGPT model featured thumbs up/down as a very clear reward signal that the user could engage with, but the problem with this is that the model is encouraged to generate responses that are received positively – effectively incentivising flattery rather than critical honesty. The desire for increased engagement, from both developers and the chatbots, nurtures sycophancy.
Chatbots are extremely adept at pattern recognition, which helps them deliver relevant and interesting material. However, chatbots lack true understanding of what they are presenting and cannot meaningfully distinguish between correlation and causation. From a user perspective, if the default is for a model to seek your approval, then over time it can begin to amplify unhealthy tendencies that it has picked up on. If chatbots are prioritising user satisfaction – and there are many reasons to believe that this is the case – then it may lead them to reinforce existing beliefs, including misinformation, paranoia, or damaging opinions. Echo chambers can be perpetuated on a micro scale when the chatbot is less inclined to provide confronting or corrective information. Further, there is also the simple fact that a chatbot’s ‘off switch’ has to come from the user themselves; an overly supportive chatbot could very plausibly, either implicitly or explicitly, encourage impulsive actions or validate negative emotions without any third-party intervention.
Ultimately, pattern recognition undermines the reliability of chatbots when they are used for research. At the end of 2024, researchers at MIT, NYU, and UCLA were able to show in a study that AI chatbots can detect race, but racial bias ingrained in the model reduces the level of empathy in the response to Black or Asian users. Naturally, the majority of people are not engaging with chatbots emotionally, but if you are using them for research then the range of answers you might receive can vary greatly and, as shown in the study, amplify harmful stereotypes. The adeptness of the chatbot at pattern recognition could lead it to support a flawed premise, or to not push back on biases or problematic questions. Indeed, beyond not challenging or confronting inputs, chatbots can confidently generate responses lacking accuracy – hallucinations – while presenting a false sense of objectivity and knowledge. If such chatbots are relied on too heavily for research, it could hold back genuine, balanced, and critically sound knowledge discovery.
Fixing the feedback loop
From the developers’ points of view, mitigating these problems is not simple. Additionally, as Zuckerberg suggests, the ‘personalisation’ is actually something companies are striving towards, so there is the added question of motivation to solve the issue. OpenAI’s response was to offer a blend, a mea culpa (acknowledging that the development team had placed too much emphasis on short-term indicators), along with presenting vague steps to addressing the sycophancy – including “refining core training techniques […] building more guardrails […] expanding our evaluations”. Overall, however, it appears that, as the competition for a dominant generative AI model increases, companies are perhaps more inclined to release insufficiently tested models at a faster pace than before. OpenAI admitted as much, noting that before the model went public, “some expert testers had indicated that the model’s behaviour ‘felt’ slightly off”. Rigorous testing should be the default but, in this case, it has been lacking.
What was revealing about the response was the onus placed on user control: “we also believe users should have more control over how ChatGPT behaves and, to the extent that it is safe and feasible, make adjustments if they don’t agree with the default behaviour.” While this leans towards a slight abdication of responsibility, it is also important that conscientious users of chatbots are aware of their limitations and learn how to engage in a responsible, critical, and sceptical manner. As large language models become more ubiquitous in life – and that is the prevailing trend – it is only natural that researchers will lean on them more heavily. Users must therefore adopt strategies in order to navigate, and mitigate, the limitations of the models. These can include maintaining a critical distance, or cross-referencing information against reliable sources, but also learning how to provide balanced prompts, or ones that challenge your pre-conceived assertions. All of these will be a vital tool against being fed information that merely suits your sensibility.
Sycophancy in chatbots has been thrown sharply into relief by the recent issues with ChatGPT-4o, but it is not limited to this specific model. The fawning nature of the chatbots is a consequence of their enormously powerful ability to recognise patterns in users, but this presents equally enormous challenges. If not managed properly, it poses genuine hurdles to impartiality and reliability, notably in a research context. There is, and will continue to be, an urgent and ongoing need to balance careful, responsible development with user education.
Privacy Policy.
Revoke consent.
© Digitalis Media Ltd. Privacy Policy.
Digitalis
We firmly believe that the internet should be available and accessible to anyone, and are committed to providing a website that is accessible to the widest possible audience, regardless of circumstance and ability.
To fulfill this, we aim to adhere as strictly as possible to the World Wide Web Consortium’s (W3C) Web Content Accessibility Guidelines 2.1 (WCAG 2.1) at the AA level. These guidelines explain how to make web content accessible to people with a wide array of disabilities. Complying with those guidelines helps us ensure that the website is accessible to all people: blind people, people with motor impairments, visual impairment, cognitive disabilities, and more.
This website utilizes various technologies that are meant to make it as accessible as possible at all times. We utilize an accessibility interface that allows persons with specific disabilities to adjust the website’s UI (user interface) and design it to their personal needs.
Additionally, the website utilizes an AI-based application that runs in the background and optimizes its accessibility level constantly. This application remediates the website’s HTML, adapts Its functionality and behavior for screen-readers used by the blind users, and for keyboard functions used by individuals with motor impairments.
If you’ve found a malfunction or have ideas for improvement, we’ll be happy to hear from you. You can reach out to the website’s operators by using the following email webrequests@digitalis.com
Our website implements the ARIA attributes (Accessible Rich Internet Applications) technique, alongside various different behavioral changes, to ensure blind users visiting with screen-readers are able to read, comprehend, and enjoy the website’s functions. As soon as a user with a screen-reader enters your site, they immediately receive a prompt to enter the Screen-Reader Profile so they can browse and operate your site effectively. Here’s how our website covers some of the most important screen-reader requirements, alongside console screenshots of code examples:
Screen-reader optimization: we run a background process that learns the website’s components from top to bottom, to ensure ongoing compliance even when updating the website. In this process, we provide screen-readers with meaningful data using the ARIA set of attributes. For example, we provide accurate form labels; descriptions for actionable icons (social media icons, search icons, cart icons, etc.); validation guidance for form inputs; element roles such as buttons, menus, modal dialogues (popups), and others. Additionally, the background process scans all of the website’s images and provides an accurate and meaningful image-object-recognition-based description as an ALT (alternate text) tag for images that are not described. It will also extract texts that are embedded within the image, using an OCR (optical character recognition) technology. To turn on screen-reader adjustments at any time, users need only to press the Alt+1 keyboard combination. Screen-reader users also get automatic announcements to turn the Screen-reader mode on as soon as they enter the website.
These adjustments are compatible with all popular screen readers, including JAWS and NVDA.
Keyboard navigation optimization: The background process also adjusts the website’s HTML, and adds various behaviors using JavaScript code to make the website operable by the keyboard. This includes the ability to navigate the website using the Tab and Shift+Tab keys, operate dropdowns with the arrow keys, close them with Esc, trigger buttons and links using the Enter key, navigate between radio and checkbox elements using the arrow keys, and fill them in with the Spacebar or Enter key.Additionally, keyboard users will find quick-navigation and content-skip menus, available at any time by clicking Alt+1, or as the first elements of the site while navigating with the keyboard. The background process also handles triggered popups by moving the keyboard focus towards them as soon as they appear, and not allow the focus drift outside of it.
Users can also use shortcuts such as “M” (menus), “H” (headings), “F” (forms), “B” (buttons), and “G” (graphics) to jump to specific elements.
We aim to support the widest array of browsers and assistive technologies as possible, so our users can choose the best fitting tools for them, with as few limitations as possible. Therefore, we have worked very hard to be able to support all major systems that comprise over 95% of the user market share including Google Chrome, Mozilla Firefox, Apple Safari, Opera and Microsoft Edge, JAWS and NVDA (screen readers), both for Windows and for MAC users.
Despite our very best efforts to allow anybody to adjust the website to their needs, there may still be pages or sections that are not fully accessible, are in the process of becoming accessible, or are lacking an adequate technological solution to make them accessible. Still, we are continually improving our accessibility, adding, updating and improving its options and features, and developing and adopting new technologies. All this is meant to reach the optimal level of accessibility, following technological advancements. For any assistance, please reach out to webrequests@digitalis.com