More generically, our species' Achilles heel is our inability to factor in the long-term cost of negative externalities when evaluating processes that yield short-term positive results.
This. From simple personal choices to the marker economy and politics. With games we're introduced to cheat codes pretty early in our lives. Some people outgrow them, some don't. Too bad our systems encourage their use, whether it's a time-to-market thing, cutting costs, or the next election.
Counterargument. The author is primarily looking at AI trend lines. Let's say our industry continues moving along alternate, equally compelling, trend lines: increasing global volatility, chaos in the energy markets, growing likelihood of great power conflict this century, climate collapse, mass migration, societal unrest, yada yada.
What happens to all of these AI-native companies if the AI bubble is not able to survive in these conditions? If your current development process is built on the metabolic equivalent of 400kg of leaves per day[0], then when the allegorical asteroid hits, you're going to be outperformed by smaller, nimbler companies with much lower resource requirements. Those companies may be better suited for survival in hostile macro conditions.
In other words, I think a lot of companies believe that they're trimming their metabolic fat by replacing engineers with AI. Lower salary costs! But at the same time, they're also increasing their reliance on brittle energy infrastructure that may not survive this century. (Not to mention the brittleness of the semiconductor fabrication pipeline, RAM availability, etc)
Predicting the future isn't about being right tomorrow, it's about selling you something today. - read that somewhere
Folks using AI aren't interested in the future, they are interested in buying today and maximizing profits today. If something goes wrong tomorrow, then that's when the problems are dealt with: tomorrow.
AI is an incredibly fragile technology, as you say, it's depended on so many things going right, amazing stuff that it works at all. That fragility includes price, once that goes up and developer price goes down, the winds of change might blow again.
AI also forces folks to be online to code, without being online, companies cannot extend their products. Git was the first version (open source) control system that worked offline. We're literally turning back the hands of time with AI.
AI is another vendor lock-in with the big providers being the sole key-holders to the gates of coding heaven. Folks are blindly running into the hands of vendors who will raise prices as soon as their investors demand their money back.
AI is "improving" code bases to make subtle errors and edge cases harder to detect debugging without using AI will be impossible. Will a human developer actually be able to understand a code base that has been coded up by an AI? That's a problem for tomorrow, today we're making the profits and pumping up the shareholder value.
AI prompts are depended on versions of LLMs - change the LLM and the prompt might will generate different code. Upgrade LLMs or change prompts and suddenly generated code degrades without warning. But prompts are single-use one-way technology: once the generated code is in the code base, there is no need for the prompt - so that's non-issue, only for auditors.
Having come from levers, to punch cards, to transistors, to keyboards, to mice and finally AI, programming has fundamentally forgotten there is a second dimension. Most fields have moved to visual representation of data - graphs, photos, images, plans etc. Programming is fundamentally a single dimension activity with lines and lines of algorithmic code. Hard to understand and harder to visualize (see UML). Now AI comes along and entrenches this dependency on text-based programming, as if the keyboard is the single most (and only) important tool for programming.
It's a lack of imagine of exploring alternatives for programming that has lead us here. Having non-understandable AI tools generating subtly failing code that we blindly deploy to our servers is not an approach that promises look term stability.
This isn't true in the broad sense you've used. It's true that most people don't have the hardware to run the bleeding-edge foundation models, but with a modest Macbook you can run very capable local models now (at least capable for coding, where my experience is).
Here I was talking of the AI vendors - they specifically provide inferior models for local usage while offering the "insanely" good models only online.
AI can be run locally but with the growth of agent factories, this is going to be less and less possible if you want to keep up with the Jones.
> AI is "improving" code bases to make subtle errors and edge cases harder to detect debugging without using AI will be impossible. Will a human developer actually be able to understand a code base that has been coded up by an AI?
Huh? It’s just code that you can read. Why do you think the code will be impregnable by a team of human minds?
Because code does not include the thought processes that went into creating the final code. Take a second and have a look at the Linux kernel code base and get into that. It's surprising how some code only make sense if you understand the bigger picture.
So it will be with AI code that has just been generated and blindly added to the code base. It makes everything work but sometimes, perhaps not always, the devil lies in the details.
Take any book, open it up to a random middle section, read it. I can read the words but I don't understand the story. And so it is with code.
So, how does aj engineer new to a code base add new features? They read the code base, understand the architecture and structure and make changes. An agent can do the same.
> Right now there are companies which hire software devs or data scientists to just solve a bunch of random problems so that they can generate training data for an LLM model.
> Which begs questions about whether closed source will provide any protection (it doesn't appear so, given how able AI tools already are at disassembly?)
Disassembly implies that you're still distributing binaries, which isn't the case for web-based services. Of course, these models can still likely find vulnerabilities in closed-source websites, but probably not to the same degree, especially if you're trying to minimize your dependency footprint.
reply