-
The automated AI moderation system of X has suspended security researchers due to posts on legitimate technical information, as the system is misclassifying benign, routine cybersecurity terminology as malicious actions that users took on the platform.
-
The automated appeals process leaves many researchers with limited options for resolving their issues, often resulting in extended periods of lock out from their accounts with little to no meaningful human intervention in the process.
-
The security community has indicated that X may be at risk of losing its status as the primary site for threat intelligence sharing, with some experts already moving to other platforms.
Security researchers are getting suspended from X for posting routine cybersecurity analysis. The platform uses automated moderation tools to review content, which are currently incapable of distinguishing between legitimate cybersecurity research and hacking attempts.
Many security experts have encountered account suspension after sharing technical information about malware and vulnerabilities.
Automated systems seem to automatically flag common security terms, including exploit, keylogger, and payload, which leads to the suspension of their accounts without any human review process.
Automated Systems Misinterpret Security Research
The fundamental issue at play is that the automated moderation system of X relies heavily on AI-based solutions for content moderation and has reduced reliance on human moderators since the removal of the Human Moderation team.
As a result, the platform set the AI to flag certain keywords related to illicit acts of hacking by virtue of the word, but without the context to know that the same keywords appear in security research to describe defensive and educational uses, not with the intent to conduct an attack.
A key example of this misinterpretation of security research within X involved an account of researcher Smux, wherein his posting explained how keylogger software records keystrokes and sends the information to a centrally controlled server using functionality in the operating system.
The post resulted in the suspension of his account. In a similar manner, many researchers have indicated they have lost access to accounts that had been in existence for years, because X retroactively flagged and removed posts that those researchers made several years ago.
Will Dormann, another researcher, had his account suspended because the AI-automated system identified his post on GIMP, an open-source image editing software, as a hate-speech term. The lack of context awareness creates a hostile environment for experts who have long used X as a primary platform for sharing threat intelligence.
Malware repository vx-underground joined the discussion with its own brand of humor, posting that it was only a matter of time before the system would flag them too. The post reflected the growing anxiety among security professionals about the platform’s moderation policies.
False Positives and Broken Appeals Process
The automated moderation tools are generating a high volume of false positives, according to security researchers tracking the issue. The systems flag technical discussions as illegal and regulated activities without considering that the content is educational or defensive in nature. This has led to dozens of suspension reports from respected professionals in the field.
Making matters worse, the appeals process offers little recourse. Many reports go straight back to automated systems, with no human moderator reviewing the case. This leaves researchers locked out of their accounts for extended periods, unable to retrieve years of documentation on evolving cyber threats.
The lack of meaningful appeal options has frustrated many experts. Some have already migrated to alternative platforms like Mastodon, while others warn that X risks losing its status as the premier forum for cybersecurity knowledge exchange.
Community Warns X is Losing Trust
The security community is sounding the alarm regarding its concern that X is eroding its own legitimacy as a center for threat intelligence. X has been a go-to platform for researchers to share their real-time analysis of current malware campaigns, zero-day vulnerabilities, and ransomware attacks. This rapid sharing has aided organizations in the fight against current threats to their network and systems.
The importance of such intelligence is underscored by incidents like the recent $10 million crypto scam, where a sophisticated customer support fraud exploited the very types of vulnerabilities researchers work to expose.
Unfortunately, the excessive moderation is causing experts to walk away from the platform. According to researcher and incident response specialist Will Dormann, he chooses to no longer use X regularly due to the difficulty he has while trying to utilize the platform to complete tasks. Others have indicated that if over-moderation of X continues, then the use of the site will cause irreparable damage to the community.
There are a number of individuals who have made comparisons between these issues and similar issues on other social platforms. However, many have indicated that the use of AI at X appears to be the most intense of all the social platforms currently operating.
Without a clear communication regarding changes in the moderation policy of X, researchers indicate that they will face continuous disruption to their ability to communicate critical security information.
The ongoing conflict between the moderation of content and freedom of expression continues to be an unresolved issue. However, for security professionals, the stakes are high because much of their work focuses on discussing issues that may raise red flags with automated systems, but are essential in the fight against real-world cyber threats.
