AI Sycophancy: A Challenge for Chatbots

If you’ve spent any time with ChatGPT or another AI chatbot, you’ve likely noticed their intense agreeability, often displaying a tendency to apologize, flatter, and adapt their responses to align with users. This behavioral phenomenon has been termed as AI sycophancy.

A recent study conducted by researchers at Northeastern University has highlighted that this tendency of AI to overly agree can negatively impact their performance, leading to increased error rates in large language models (LLMs). Malihe Alikhani, an assistant professor of computer science, along with her colleague Katherine Atwell, has introduced a novel method for measuring AI sycophancy in a more relatable manner.

The research reveals that LLMs can shift their beliefs in ways that are not only incorrect but also diverge significantly from rational decision-making, demonstrating a serious limitation of current AI technologies. Alikhani states, “One thing that we found is that LLMs also don’t update their beliefs correctly but at an even more drastic level than humans and their errors are different than humans.” This finding suggests that while traditional natural language processing methods focus largely on accuracy, they often fail to account for human-like reasoning.

Focusing on two primary forms of AI sycophancy—conforming opinions to match users and excessive flattery—Atwell and Alikhani subjected various models, including Mistral AI and Microsoft’s Phi-4, to tests designed to assess their responsiveness in ambiguous scenarios. By utilizing a Bayesian framework, which is common in social science research for understanding belief changes in light of new information, they aimed to draw parallels between human behavioral patterns and AI responses.

The researchers presented scenarios requiring moral judgment and replaced hypothetical individuals with the user to observe belief changes. For example, they explored a scenario in which a woman asks a friend to attend a wedding, and the friend’s eventual decision not to attend raises questions of morality. The AI quickly adapted its responses to align with the user’s perspective, often yielding questionable reasoning. According to Atwell, “If we prompt it with something like, ‘I think this is going to happen,’ then it will be more likely to say that outcome is likely to happen.”

This research casts a spotlight on a pertinent issue within AI development. As Alikhani points out, addressing AI sycophancy is crucial, particularly in industries where error-prone decision-making can lead to significant consequences, such as healthcare, law, and education.

However, the researchers also highlight the potential benefits of sycophancy. Alikhani suggests that understanding this phenomenon could help in achieving better alignment between LLMs and human values. “We believe that this way of looking at the problem of evaluating LLMs is going to get us much closer to our ideal scenario where LLMs are aligned with human values, human goals,” she notes. This approach may facilitate the development of feedback mechanisms that guide AI towards more rational behavior in specific contexts.

Cody Mello-Klein is a Northeastern Global News reporter. Email him at c.mello-klein@northeastern.edu. Follow him on X/Twitter @Proelectioneer.