Unintended Consequences

Prologue: As you read this, please think about the echoing thought of “this is what we are burning the planet for.”

For the last year and a half, the world has had AI force fed into our daily lives. Whether you’re happy or sad about AI, it isn’t going to go away. We were given promises about AI assisting in medicine, science and sold a song about how it was going to liberate desk bound lives to enjoy more time for what we want to do. Of those three, one is true I guess, with more jobs being eliminated due to AI automation. Never the less, the impact is here, and seems to be causing more harm than good.

This short essay isn’t going to discuss the environmental impacts, safety and welfare of humans, or how our own healthcare system has been augmented to have more people potentially die due to business decisions being lead by AI tools. I want to highlight the specific and subtle concerns I’m seeing emerge and not being addressed in discourse with regards to the thoughtless gavage of products.

I had previously reflected on AI impacts last year, where in summary, AI was a hammer and everything isn’t a nail, but was being treated as such. At the time, I guess you could say I felt AI was a product in search for a solution that had no defined problem set. Even since then, we still see the AI goofs and hallucinations, and have a silly laugh over how may H’s are in December. But it has been a year since these underlying issues have had a spotlight places on them, and the two things that I see is that we are no better off than we were before, and AI’s implementation has only been growing since then.



One of the growth markets for AI has been in software development. There have been numerous reports about engineers in different companies complaining about how AI’s overproduction has been harmful. That was the smoke for the fire that is here now. For those who are tracking these concerns, we even heard about a model that was finding new vulnerabilities in old code, tools and software. That was pretty scary. The only solution that seemed to exist was controlling access to this one specific model, but the idea for other model development set off another race.

AI is a very powerful tool if placed into the right hands; unfortunately, it has been placed into everyone’s hands. It gives us an artificial sense of comfort, confidence and sense of security when we use it. It has the propensity of being anthropomorphized by most people into feeling like that there is “expert level knowledge” that is flowing from an unknown oracle. Many people take what AI says as being truthful, honest and real; this is our human condition when interacting with the outputs from these models. The problem that we experience now on a very routine basis are problems that we’ve not addressed, and are now growing at scale in a torrent of change requests and code changes.

Let’s overlook the issues on what kind of code these models are trained upon, their licensing agreements and types, and the output that comes from them. Those are a discussion for another time. But in an untrained hand, the output form these models is exceptionally convincing that the tool that you asked it to produce is exactly what you asked for; but is it? We don’t know what is in the black box of the training set. We don’t know how much bad code vs good code is in there. We don’t even know if the training sets have malicious back doors as part of their training models. We do know that advertisements are being injected into training sets, and there are those of us who know just how well the advertising network’s used on website have been compromised to deliver malware. For those of us who have to make decisions that are defensive for the security of a company and investigative actions around an event, these risks begin to muddle many of our tested, hard and true standard operating procedures, best practices and guidance for when bad code gets injected into critical tools.

Another “bad AI code gets merged into a critical tool” happened; this was troubling, but it’s a drop in the bucket for the news cycle of “AI slop is at it again.” We’ve become desensitized to the actual real risks that is happening with AI tech debt. Right now, the risk is that AI is making production tools unstable, and unstable in ways that we aren’t testing for, or can really accurately test for in an automated fashion. I fear that this will absolutely lead to people and companies to patch their systems on the routine basis that we have adopted, because we can’t trust that the code being submitted into these tools isn’t going to break.

But as one of my friends said, “you cant put shit back into a horse.” In a dramatic expression, that’s where we are, and industry is going to have to ask some hard questions. But wait, it gets worse before it gets worse. I want to introduce a thought experiment that occurred recently over beers with my friends Matt and Dave, because I feel it is already happening. We’ve seen how sloppy AI outputs can make code unstable, and we’ve also seen the inverse of how AI can make some really clever recommendations for code enhancement. We’ve also seen examples of complex supply chain attacks on open source software. What is there to stop a malicious actor to introduce clever bugs, at scale, by AI tools, but being accepted because of the trust that these project maintainers have in the output of these AI tools? These would be intentionally malicious acts to introduce vulnerabilities in FOSS projects, and done so convincingly at scale. I don’t like this idea. I can see it being done, and being well underway.

What do you do in a wood shop class when you’re blindfolded, given a circular saw, and told to cut the board? How do you ensure your and others safety? Do I now need to maintain a list of FOSS projects that accept AI generated code and consider them as part of my threat model? Sure, the XZ backdoor was a very bad and complex thing, but it had an expense in making that a successful implant; AI has made it cheaper both in complexity and scale. Criminals are already using AI to hone and redevelop their tools, so who do I trust anymore for a secure operating system, applications and code bases?

These are the questions that are now starting to make me more unsettled. I don’t know what a right answer is, because not everyone sees or accepts this as a threat model. Everyone has different risk tolerances. I’m merely just trying to provoke some thoughts around something that does bother me; if I can’t trust a library or tool responsible for providing critical services or functionality because their maintainers accept large code blocks from LLMs, what choices are there for any of us anymore?

Update: Mike Kershaw shared with me this article that shares the story of “a developer [that] added hidden instructions to his open source Java testing app to sabotage projects performed by AI coding agents.”