AI Bots Manipulate Beliefs – Endangering Democracy
The Rise of AI-Powered Social Bots
In mid-2023, during a period when Elon Musk rebranded Twitter as X and before he ended free academic access to the platform’s data, my colleagues and I were investigating signs of social bot accounts that used artificial intelligence to generate content. These bots are AI software designed to post content and interact with users on social media. Our research uncovered a network of over a thousand bots involved in crypto scams. We named this network “fox8” after one of the fake news websites it aimed to promote.
We were able to detect these accounts because the coders made some mistakes: they occasionally missed posts that revealed their true nature, such as those containing self-revealing text from ChatGPT. For example, the AI model sometimes refused to comply with prompts that violated its policies. The most common response was, “I’m sorry, but I cannot comply with this request as it violates OpenAI’s Content Policy on generating harmful or inappropriate content. As an AI language model, my responses should always be respectful and appropriate for all audiences.”
We believe that fox8 was just the beginning of a larger issue. Better coders could filter out self-revealing posts or use open-source AI models that have had ethical guardrails removed.
Coordinated AI Bots and Their Impact
The fox8 bots created fake engagement by interacting with each other and human accounts through realistic back-and-forth discussions and retweets. This tactic tricked X’s recommendation algorithm into amplifying their posts, leading to significant follower growth and influence. Such coordinated efforts among inauthentic online agents were unprecedented—AI models had been weaponized to create a new generation of social agents that were far more sophisticated than earlier bots. Machine-learning tools to detect social bots, like our own Botometer, struggled to differentiate between these AI agents and real human accounts. Even AI models trained to detect AI-generated content failed to identify them.
The Era of Generative AI Bots
Fast forward a few years: Today, people and organizations with malicious intent have access to more powerful AI language models, including open-source ones. Meanwhile, social media platforms have relaxed or eliminated moderation efforts. They even offer financial incentives for engaging content, regardless of whether it is real or AI-generated. This creates a perfect storm for foreign and domestic influence operations targeting democratic elections. For instance, an AI-controlled bot swarm could create the false impression of widespread, bipartisan opposition to a political candidate.
The current U.S. administration has dismantled federal programs that combat such hostile campaigns and defunded research efforts to study them. Researchers no longer have access to the platform data needed to detect and monitor these kinds of online manipulations.
Interdisciplinary Efforts to Address the Threat
I am part of an interdisciplinary team of computer science, AI, cybersecurity, psychology, social science, journalism, and policy researchers who have raised alarms about the threat of malicious AI swarms. We believe that current AI technology allows organizations with malicious intent to deploy large numbers of autonomous, adaptive, and coordinated agents across multiple social media platforms. These agents enable influence operations that are far more scalable, sophisticated, and adaptive than simple scripted misinformation campaigns.
Rather than generating identical posts or obvious spam, AI agents can produce varied, credible content at scale. The swarms can send tailored messages to users based on their preferences and the context of their online conversations. They can adjust tone, style, and content dynamically in response to human interaction and platform signals like likes or views.
Synthetic Consensus and Psychological Manipulation
In a study conducted last year, we used a social media model to simulate swarms of inauthentic social media accounts using different tactics to influence a target online community. One tactic proved particularly effective: infiltration. Once an online group is infiltrated, malicious AI swarms can create the illusion of broad public agreement around the narratives they are programmed to promote. This exploits a psychological phenomenon known as social proof: humans tend to believe something if they perceive that “everyone is saying it.”
Such astroturf tactics have existed for years, but malicious AI swarms can now create believable interactions with targeted users at scale, getting them to follow the inauthentic accounts. For example, agents can discuss the latest game with a sports fan or talk about current events with a news junkie. They can generate language that resonates with the interests and opinions of their targets.
Even if individual claims are debunked, the persistent chorus of independent-sounding voices can make radical ideas seem mainstream and amplify negative feelings toward “others.” Manufactured synthetic consensus poses a real threat to the public sphere, the mechanisms democratic societies use to form shared beliefs, make decisions, and trust public discourse. If citizens cannot distinguish between genuine public opinion and algorithmically generated simulation of unanimity, democratic decision-making could be severely compromised.
Mitigating the Risks
Unfortunately, there is no single solution. Regulation granting researchers access to platform data would be a critical first step. Understanding how swarms behave collectively is essential to anticipate risks. Detecting coordinated behavior is a key challenge. Unlike simple copy-and-paste bots, malicious swarms produce varied output that resembles normal human interaction, making detection much more difficult.
In our lab, we design methods to detect patterns of coordinated behavior that deviate from normal human interaction. Even if agents look different from each other, their underlying objectives often reveal patterns in timing, network movement, and narrative trajectory that are unlikely to occur naturally.
Social media platforms could use such methods. I believe that AI and social media platforms should also more aggressively adopt standards to apply watermarks to AI-generated content and recognize and label such content. Finally, restricting the monetization of inauthentic engagement would reduce the financial incentives for influence operations and other malicious groups to use synthetic consensus.
The Threat Is Real
While these measures might mitigate the systemic risks of malicious AI swarms before they become entrenched in political and social systems worldwide, the current political landscape in the U.S. seems to be moving in the opposite direction. The Trump administration has aimed to reduce AI and social media regulation and instead favors rapid deployment of AI models over safety.
The threat of malicious AI swarms is no longer theoretical: our evidence suggests these tactics are already being deployed. I believe that policymakers and technologists should increase the cost, risk, and visibility of such manipulation.
Leave a Reply