Python security developer-in-residence decries use of bots that ‘cannot understand code’
Software vulnerability submissions generated by AI models have ushered in a “new era of slop security reports for open source” – and the devs maintaining these projects wish bug hunters would rely less on results produced by machine learning assistants.
Seth Larson, security developer-in-residence at the Python Software Foundation, raised the issue in a blog post last week, urging those reporting bugs not to use AI systems for bug hunting.
“Recently I’ve noticed an uptick in extremely low-quality, spammy, and LLM-hallucinated security reports to open source projects,” he wrote, pointing to similar findings from the Curl project in January. “These reports appear at first glance to be potentially legitimate and thus require time to refute.”
Larson argued that low-quality reports should be treated as if they’re malicious.
As if to underscore the persistence of these concerns, a Curl project bug report posted on December 8 shows that nearly a year after maintainer Daniel Stenberg raised the issue, he’s still confronted by “AI slop” – and wasting his time arguing with a bug submitter who may be partially or entirely automated.
In response to the bug report, Stenberg wrote:
We receive AI slop like this regularly and at volume. You contribute to [the] unnecessary load of Curl maintainers and I refuse to take that lightly and I am determined to act swiftly against it. Now and going forward.
You submitted what seems to be an obvious AI slop ‘report’ where you say there is a security problem, probably because an AI tricked you into believing this. You then waste our time by not telling us that an AI did this for you and you then continue the discussion with even more crap responses – seemingly also generated by AI.
Spammy, low-grade online content existed long before chatbots, but generative AI models have made it easier to produce the stuff. The result is pollution in journalism, web search, and of course social media.
For open source projects, AI-assisted bug reports are particularly pernicious because they require consideration and evaluation from security engineers – many of them volunteers – who are already pressed for time.
Larson told The Register that while he sees relatively few low-quality AI bug reports – fewer than ten each month – they represent the proverbial canary in the coal mine.
“Whatever happens to Python or pip is likely to eventually happen to more projects or more frequently,” he warned. “I am concerned mostly about maintainers that are handling this in isolation. If they don’t know that AI-generated reports are commonplace, they might not be able to recognize what’s happening before wasting tons of time on a false report. Wasting precious volunteer time doing something you don’t love and in the end for nothing is the surest way to burn out maintainers or drive them away from security work.”
Larson argued that the open source community needs to get ahead of this trend to mitigate potential damage.
“I am hesitant to say that ‘more tech’ is what will solve the problem,” he said. "I think open source security needs some fundamental changes. It can’t keep falling onto a small number of maintainers to do the work, and we need more normalization and visibility into these types of open source contributions.
“We should be answering the question: ‘how do we get more trusted individuals involved in open source?’ Funding for staffing is one answer – such as my own grant through Alpha-Omega – and involvement from donated employment time is another.”
While the open source community mulls how to respond, Larson asks that bug submitters not submit reports unless they’ve been verified by a human – and don’t use AI, because “these systems today cannot understand code.” He also urges platforms that accept vulnerability reports on behalf of maintainers to take steps to limit automated or abusive security report creation.
Our security@ address at $dayjob gets about that many a month. Lots of folks blindly sending bug reports and “politely requesting a finder’s fee for disclosing properly.”
The shit of it is, they’ll all for stuff we don’t even use. IIS vuln reports when we only use Apache. Stuff like that.
So, this isn’t quite the issue being raised by the article – that’s bug reports generated on bug trackers by apparently a bot that they aren’t running.
However, I do feel that there’s more potential with existing LLMs in checking and flagging potential errors than in outright writing code. Like, I’d rather have something like a “code grammar checker” that highlights potential errors for my examination rather than something that generates code from scratch itself and hopes that I will adequately review it.
I’d rather have something like a “code grammar checker” that highlights potential errors for my examination rather than something that generates code from scratch itself
Agreed. The other good use case I’ve found is as a faster reference for simple things. LLMs are absolutely great for one-liners and generating troublesome (but logically simple) things like complex xpath queries. But I still haven’t seen one generate a good script of even moderate complexity without hand-holding. In some cases I’ve been able to get usable output with a few shots, saving me a bit of time compared to if I’d written the whole darned thing from scratch.
I’ve found LLMs very useful for coding, but they aren’t replacing my actual coding, per se. They replace looking things up, like through man pages, language references, or StackOverflow. Something like ffmpeg, for example, has a million options and it is always a little annoying to sift through the docs manually when I just need to do one specific task.
I’m sure it’ll happen sooner or later. I’m not naive enough to claim that “computers will never be able to do $THING” anymore. I’ll say “not in the next year”, though.
Your and @tal@lemmy.today’s experiences are basically the same as mine. Except with translation instead of programming.
only for the publication to use AI to generate a stupid image for their article thumbnail.
I don’t think that’s AI
Some would say that this is a tactic by entities like the NSO Group to drown real vulnerabilities in spam.
It’s not unlikely. Not unlikely at all.
Larson argued that low-quality reports should be treated as if they’re malicious.
It’s refreshing and uplifting to see this sort of sanity.
So people are asking LLMs to come up with a problem that they can then file?
There are a number of jobs that count various community contributions in your review. This is a very large topic - write a paper in a major journal, give a speech at a conference, submit high profile bug reports - you can get a large raise for it. (the worry of course is that you get a reputation and thus someone gives you a great offer so if they want to keep you they better pay enough that you won’t leave). Exactly what gets you those promotions/raises isn’t clear in part because they need flexibility for someone who honestly discovers a new way to get that reputation and thus they have to give them a promotion. People who don’t deserve the promotion see the policy in place and look for ways to cheat themselves to a promotion they don’t deserve.
so, the problem is once again capitalism in disguise…
Do you really think that had AI been available to apparatchiks in Communist countries, they wouldn’t have used it to advance their careers?
The problem isn’t capitalism, it’s human nature, regardless of the system. Incentivize behavior that is beneficial to the individual (even if just in the short term), but not society as a whole and people will engage in it. It doesn’t matter if there’s a democratically elected leader, monarch or first party secretary at the helm of the nation.
This type of parasitic, even sociopathic behavior is directly rewarded in capitalism, though. Kinda figure that’s all they meant.
Also, it capitalism is anywhere, it’s everywhere. Or this is at least true as long as the United States is in the picture…
Do you blame capitalism and America for bad weather too - or when you stab your toe in the morning?
Capitalism is a product of human nature; nobody designed it that way. When people attempt to design better systems from the ground up, far worse human behavior is being directly rewarded. Seriously, do you have any idea how much more disgustingly selfish and self-centered people are under economic and political systems that are supposedly better?
If you look at the most democratic nations on Earth, the ones with the best functioning institutions, the best education, the most innovation, least inequality, you’ll find nations that are fiercely capitalist, with strong mercantile tradition dating back centuries. These people were capitalists before the term was first coined and they selfishly wanted the state to protect their investments, so they created strong institutions for that purpose. They had no idea that these institutions would end up doing so much more, spreading and maintaining wealth far beyond the small elite that they were supposed to serve while at the same time slowly moving power away from them. The many smaller educated merchants, who only educated themselves, because they selfishly wanted more prosperity for themselves, ended up being an amazing nucleus of a well-formed civil society, which is the backbone of every single successful free country.
Forget about America for a second or pie in the sky ideas that failed spectacularly any time they came in contact with the basic reality of human nature. This is what works: Stumble into a system that accidentally rewards selfish human behavior in such a way that everyone ends up benefiting from it. The problem from the perspective of ideologues is that this isn’t glamorous, there are no dashing revolutionaries applying catchy slogans with the butts of their rifles. It’s slow, incredibly difficult to replicate, requires rewarding the “wrong” kind of people for the longest time and. There’s no trickling down or other such nonsense, but rather the slow collective realization that the same system that protects investments and the free exchange of goods and services can do a rather excellent job at protecting and increasing civil rights. It was neither linear nor planned and the resulting societies are by no means perfect, but they are the best we managed to achieve as a species so far, so consider learning from them how they were able to make capitalism work.
Sorry for the uncalled for wall of text, but I’m increasingly tired of people here blaming capitalism for everything. It comes across as performative, even downright intellectually lazy. I get that this is a left-leaning place to say the least and there’s a reason why I’m here too, because I’m identifying with many typical left political positions - but certainly not all of them and most definitely not those that have failed historically and don’t hold up to the most basic of scrutiny.
Hm, I totally get how you are frustrated with people using one-dimensional, overly used and in-group accepted answers to respond to very complex questions. Yes, they can feel pretty performative at times. “Capitalism bad” is an easy way to respond to all kinds of problems and not always useful. I guess I can also understand people using “capitalism bad” as an answer because after analyzing capitalism and all the consequences stemming from worldwide capitalist domination, it gets really frustrating always having to answer with a well thought out analysis. So a shorthand like “capitalism bad” can be quite handy.
Regarding your comment, you seem like a person that believes many of the capitalist talking points. (It reads a bit like Sabine Hossenfelder’s video on capitalism).
First of all, you talk about inequality and how capitalist societies have higher equality. You realize how nonsensical this is regarding how the most powerful, rich people are coming from the capitalist country that has hardly any healthcare and people living in incredible poverty, right? If you look at a list of inequalities worldwide based on GDP, you can see that the US files on rank C, behind the Philippines, Pakistan, UAE and Russia (just to name a few). And choosing inequality is in itself probably a rather bad measure because rich capitalist countries have been oppressing, exploiting and destroying other countries to maximize their own profits for as long as capitalism exists. And this is were the myth of capitalist countries bettering the lives of people stems from. Of course people of all classes are more wealthy in countries that exploit other countries. Comparing countries in isolation then is like a faulty equation where you leave out how country A is actually robbing country B. And this isn’t only true for the US alone, but for the whole Global North. I’m from Germany and this country’s riches are solely possible on the backs of slave workers around the world. Classism isn’t local, country-based anymore, we have found even lower classes of people to exploit. Colonialism is still running the world, but now in a new design.
Even if we don’t stay at global or country level but zoom in a bit you’ll find that technology and progress is often made not because but despite of capitalism. Uncontrolled capitalism does not work in favor of people. People have to intervene and contain it all the time. Look at the pharma industry for example. Look at patents, like for important vaccines, agricultural technologies or really anything else. Look at companies giving a shit about their worker’s health (or their human rights) or the environment. All of this behavior is rewarded in a capitalist system because it is about maximizing profits and accumulating wealth alone. Sure, there are some light versions of capitalism like social market economy (like in Germany a few decades ago). But again, this is still based on exploiting people and keeping them poor.
And are you serious about the civil rights being a by-product of capitalism? Again, civil rights have happened despite capitalism. It has been grassroot movements and anticapitalists that have been marching in the streets fighting for civil rights for the most time. Capitalism in itself just doesn’t care for human rights at all, there is no advantage to them. On the opposite, patriarchy is a by-product of capitalism giving it even more control over people and maximizing the work force. Civil rights in capitalist countries may be more advanced not because of a capitalist system but because these countries are much richer. Again, because they exploit everyone else! People like us in rich countries having civil rights have caused many people in other countries to have no civil or human rights, all the time. Rich countries and companies may have civil rights at home, but they really don’t care about supporting dictators, fascist movements or discriminatory practices elsewhere. On the contrary, keeping a dictator in place is much better for maximizing your profits because you have much better control over that country. Our rich countries have an incentive to keep civil and human rights low in other countries because of capitalist logic.
Regarding what you say about capitalism not containing any “dashing revolutionaries or catchy slogans” I partially agree with you. There are certainly people that made themselves comfortable in this niche of glorifying communism or any other revolutionary movement but that secretly do not want to change anything at all. But this is then only a critique of these few people and it adds nothing to the debate at all. It actually seems more like a straw man argument by you to defend capitalism. Who said socialism, anarchy, whatever needed any “dashing revolutionaries or catchy slogans”?
