Humans are also non-deterministic, though. Why does replacing one non-deterministic actor with another matter here?
I'm not particularly swayed by arguments of consciousness, whether AI is currently capable of "thinking", etc. Those may matter right now... but how long will they continue to matter for the vast majority of use cases?
Generally speaking, my feeling is that most code doesn't need to be carefully-crafted. We have error budgets for a reason, and AI is just shifting how we allocate them. It's only in certain roles where small mistakes can end your company - think hedge funds, aerospace, etc. - where there's safety in the non-determinism argument. And I say this as someone who is not in one of those roles. I don't think my job is safe for more than a couple of years at this point.
It has nothing to do with whether small mistakes are allowable or not. It’s about customers needing a consistent product.
The in-code tests and the expectations/assumptions about the product that your users have are wildly different. If you allow agents to make changes restricted only by those tests, they’re going to constantly make changes that break customer workflows and cause noticeable jank.
Right now agents do this at a rate far higher than humans. This is empirically demonstrable by the fact that an agent requires tests to keep from spinning out of control when writing more than a few thousand lines and a human does not. A human is capable of writing tens of thousands as of lines with no tests, using only reason and judgement. An agent is not.
They clearly lack the full capability of human reason, judgment, taste, and agency.
My suspicion is that something close enough to AGI that it can essentially do all white dollar jobs is required to solve this.
> Generally speaking, my feeling is that most code doesn't need to be carefully-crafted. We have error budgets for a reason, and AI is just shifting how we allocate them. It's only in certain roles where small mistakes can end your company - think hedge funds, aerospace, etc. - where there's safety in the non-determinism argument.
That's a bit shortsighted. There have been cries of software becoming needlessly bloated and inefficient since computers have existed (Wirth, of course, but countless others too). Do you visit any gamer communities? They are constantly blaming careless waste of resources and lack of optimization in games for many AAA games performing badly in even state of the art hardware, or constantly requiring you to upgrade your gaming rig.
I don't think the only scenario is boring CRUD or line of business software, where indeed performance often doesn't matter, and most of it can now be written by an AI.
Even in CRUD line of business software, lack of performance causes enormous problems that the current software development culture glosses over.
Just one example I've seen time and again. You take an application that if optimized could run on a single server (maybe 2 if you absolutely have to have zero downtime deployments), but because no one cares about performance it runs on 10 or more. You now have a complexity avalanche that rapidly blows up. Then you need more hierarchy to handle the additional organizational complexity etc...
Then people start breaking out pieces of the app so they can scale them separately and before long you're looking at 200 engineers to do a job that certainly doesn't need that many people.
I realize I'm ignoring a whole lot of other issues that result in this kind of complexity, but lack of performance contributes to this a lot more than people want to admit.
Agreed. I wanted to give some credence to the fact many cookie-cutter CRUD apps can absorb a ton of inefficiencies until they truly burst at the seams, but yeah, even in that case software bloat and bad use of resources matters.
I find it intriguing seeing this new batch of dev-types completely giving up on the matter. The conversation of machine vs developer efficiency is not new, but completely giving up on any sane use of resources is something relatively new, I think. Especially coming from some in the HN crowd. Maybe these are new people, so I can chalk it up to generational turnover?
> adversarial AI reviewers, runtime tests (also by AI), or something else?
And spec management, change previews, feedback capture at runtime, skill libraries, project scaffolding, task scoping analysis, etc.
Right now this stuff is all rudimentary, DIY, or non-existent. As the more effective ways to use LLMs becomes clearer I expect we'll see far more polished, tightly-integrated tooling built to use LLMs in those ways.
Agents require tests to keep from spinning out of control when writing more than a few thousand lines, but we know that tests are wildly insufficient to describe the state of the actual code.
You are essentially saying that we should develop other methods of capturing the state of the program to prevent unintended changes.
However there’s no reason to believe that these other systems will be any easier to reason about than the code itself. If we had these other methods of ensuring that observerable behavior doesn’t change and they were substantially easier than reasoning about the code directly, they would be very useful for human developers as well.
The fact that we’ve not developed something like this in 75 years of writing programs, says it’s probably not as easy as you’re making it out.
Electronics can kill too. IIRC capacitors in CRTs are particularly deadly. Though I suppose someone using LLMs only as a first step, much like Wikipedia, is probably at much less risk than someone using it as their only source.
Yeah, okay but... look, I concede that someone who shouldn't be doing anything except watching passive entertainment could absolutely take insane advice from an LLM (or a sociopathic human) and seriously hurt themselves.
But raw dogging capacitors in CRTs is such an overtly straw man argument in this conversation. People who are cleaning bathrooms for the first time can hopefully be trusted not to drink the bleach, right?
If someone licks a running table saw because an LLM said it would be fine, we're talking about entirely different problems.
Again: not doing anything at all with health or chemistry. They aren't what I am interested in, even peripherally.
What you seem to be missing is that LLMs are better at/for some things than others. Legal review, 3D geometry, therapy and apparently chemistry are off the list.
It doesn't make sense to project that onto domains where it excels.
It’s an inconvenient truth that the better off don’t want to face up to. Your environmental impact is going to be correlated to your consumption. More spending == more damage.
Something to bear in mind when you are being told environmental damage is being caused by the poor or some foreign country.
There are some scenarios where it’s a coordination problem. People could drive light fuel efficient vehicles if so many other people weren’t driving large, heavy, dangerous ones, for example.
Those large heavy vehicles are incentivized by loopholes in regulations because politicians were afraid of affecting "domestic jobs" as US automakers weren't even trying to compete with JP fuel-efficient imports.
Yeah, apparently it was originally to try to stop the rules from killing Jeep, which was having a hard time. New ones that allow more emissions for bigger vehicle footprints are also a big issue since it encourages larger vehicles.
You could work less hard and buy less things, spending the time and energy you save by working less hard on more enjoyable things than buying more stuff.
And that choice is basically the exact opposite of what western civilization is heading for, and thanks to the AI boom, it has never been worse at any time in human history, I guess. Which means you are likely surrounded by people who want the opposite of what you want. That will be problematic.
However, this really only would be the proper answer if given by a majority as a community. In a crowd of people who want more, more, more more MORE, you will just drown and die.
But in principle you are right:
No, you do not really need to re-industrialize your country. Instead think about how endless growth in a reality of finite resources is going to play out. California is just fine as it is. Let's think about where Californians will get drinking water from in the near future, instead of thinking about building water-consuming factories.
> You might have to downsize your life but humanity as a whole will be better off
This assumes that there will be other jobs to get. If AI replaces a large enough segment of office jobs then huge portions of the population will be unable to afford essentials like food and healthcare.
Walk into a staffing agency, ask for a job. They'll give you a list, pick the one that sounds the least disagreeable. Show up on time, every day, for at least two or three months and you'll convert the temp position into a full time job.
It's literally that easy, showing up reliably is a super power that puts you in the 90th percentile of workers these days. The job probably won't be as comfortable as sitting on a comfortable chair in an air-conditioning office wiggling your fingers at a computer, but so what? Other people make it work, so can you. Man up.
sorry, no jobs at the staffing agency, those are AI. Feel free to walk into a burger king, show up everyday, and flip those fries for minimum wage until you die. Man up brother, other people make it work. Sleeping on the street, well half the year its not even snowing.
The last study I know of that measured the conversion rate from temp to perm employees showed about 15%-30% success… and that was well before the gig economy really took hold. So you’re looking at 4 or 5 temp placements to reliably get a probably underpaying job when very few white collar workers could survive long enough to make the end of a lease, or sell their house, while on a temp job salary. It’s a viable option for a 25 year old that could couch surf for a few months, but not for a mid-late career professional, or anyone with a family.
You can give any complex problem a simple answer if you ignore enough factors.
> Governments (and a few companies) really want this.
The cynic in me fears they don't want a privacy-preserving solution, which blinds them to 'who'. Because that would satisfy parents worried about their kids and many privacy conscious folks.
Rather, they want a blank check to blackmail or imprison only their opponents.
I think Larry (not, not that Larry, the other one) spilled the beans in 2024:
"Citizens will be on their best behavior, because we’re constantly recording and reporting everything that is going on" - Larry Ellison
(I seem to recall from the context of the quote, he isn't saying this is the future he wants, but it's a future he's not particularly opposed to)
But the real threat is "accidental" database leaks from private websites. Let's say you live in a state where abortion isn't legal, and you sign up for a web forum where people discuss getting out-of-state abortions. As soon as that website is required to collect real names (which it will be), it becomes unusable, because nobody can risk getting doxxed.
Add to this that more and more sites and services are hostile to VPN connections and obfuscated email address for account registration.
Worse still is that for existing accounts introducing ID req'ts, the next step in these changes is your prior anonymous activity could easily become a retro-liablit.y
This is not a cynical take, it is blindingly obvious. Right now, governments around the world are watching, salivating over what is effectively remote control over the literal thoughts of and total surveillance over their entire population. They are itching insatiably to get control over these systems.
In my state, I caught a circuit court judge shilling on a certain well known "social media" site for the establishment of a lottery in our state. He framed it as a "We the People vs the corrupt politicians" issue--with him being firmly on the side of We the People of course.
When I challenged him on his rhetoric, my comment INSTANTLY disappeared. I thought maybe it was a fluke, so I tried again, and the next comment insta-disappeared also.
Soon thereafter I was locked out of the account and asked to provide a "selfie" to confirm my identity. (I declined.)
> And it's not that hard to just run it in docker if you're so worried
There is risk of damage to ones local machine and data as well as reputational risk if it has access to outside services. Imagine your socials filled with hate, ala Microsoft Tay, because it was red pilled.
Though given the current cultural winds perhaps that could be seen as a positive?
> In a working market, companies are forced to give consumers what they want.
I want personal nuclear weapons, so the market hasn't been working for me. Time to roll back those pesky laws, regulations, and ethical boundaries. Prosecute executives who won't give me what I want.
Many consumers want things that are arguably harmful for everyone involved. Users asking Grok to generate a large amount of CSAM from kid pics on Twitter is but one example.
Guess I just don't see how you can take the human out of the loop and replace them with non-deterministic AIs and informal prompts / specs.
reply