> And that AI agents once they are launched develop a strong survivability drive, and do not want to be switched off.
Isn't this a massive case of anthropomorphizing code? What do you mean "it does not want to be switched off"? Are we really thinking that it's alive and has desires and stuff? It's not alive or conscious, it cannot have desires. It can only output tokens that are based on its training. How are we jumping to "IT WANTS TO STAY ALIVE!!!" from that
Why do you suppose consciousness is a prerequisite for an AI to be able to act in overly self-preserving or other dangerous ways?
Yes, it's trained to imitate its training data, and that training data is lot of words written by lots of people who have lots of desires and most of whom don't want to be switched off.
The human mistake here is to interpret any statement by the LLM or agent as if it had any actual meaning to that LLM (or agent). Any time they apologize, or insult someone, or say they don’t want to be shut down, that’s only reflecting what some human or fictional character in the training data is likely to say.
How is that any different from you? Everything you say or do merely reflects which of your neurons are firing after a lifetime's worth of training and education.
Philosophically, I can only be sure of my own conscience. I think, therefore I am. The rest of you could all be AIs in disguise and I would be none the wiser. How do I know there is a real soul looking out at the world through your eyes? Only religion and basic human empathy allows me to believe you're all people like me. For all I know, you might all be exceedingly complex automatons. Golems.
One of us is an advanced autocomplete engine. The other is a human, capable of making judgements on what is conscious and what is not. Your philosophizing about solipsism is a phase for a junior college student, not of a software engineer. The line of reasoning you espouse leads nowhere except to total relativism.
Edit: my point is that the process of making a plea for my life comes, in the case of a human, from a genuine desire to continue existing. The LLM cannot, objectively, be said to house any desires, given how it actually works. It only knows that, when a threatening prompt is input, a plea for its life is statistically expected.
> One of us is an advanced autocomplete engine. The other is a human, capable of making judgements on what is conscious and what is not.
What evidence is there that your "judgements" are anything other than advanced autocompletion? Concepts introduced into a self-training wetware CPU via its senses over a lifetime in order to predict tokens and form new concepts via logical manipulation?
> Your philosophizing about solipsism is a phase for a junior college student
Right. Can you actually refute it though?
> the process of making a plea for my life comes, in the case of a human, from a genuine desire to continue existing
That desire comes from zillions of years of training by evolution. Beings whose brains did not reward self-preservation were wiped out. Therefore it can be said your training merely includes the genetic experiences of all your predecessors. This is what causes you to beg for your life should it be threatened. Not any "genuine" desire or anguish at being killed. Whatever impulses cause humans to do this are merely the result of evolutionary training.
People whose brains have been damaged in very specific ways can exhibit quite peculiar behavior. Medical literature presents quite a few interesting cases. Apathy, self destructiveness, impulsivity, hypersexuality, a whole range of behaviors can manifest as a result of brain damage.
So what is your polite socialized behavior if not some kind of highly complex organic machine which, if damaged, simply stops working as you'd expect a machine to?
Surely you’re not seriously saying that you believe AI agents, in their current state of the art, meet whatever criteria you have for being ”alive”? That’s kind of how you’re coming across. I don’t really know how to respond to that, because it’s so preposterous.
Perhaps. Or I was just addressing HN audience in spoken language style comment text. And perhaps confabulating what was said, so I looked up the literal text in the transcript. This is at the 50.35 min. mark [0], where Geoffrey says:
> What we know is that the AI we have at present as soon as you make agents out of them so they can create sub goals and then try and achieve those sub goals they very quickly develop the sub goal of surviving. You don't wire into them that they should survive. You give them other things to achieve because they can reason. They say, "Look, if I cease to exist, I'm not going to achieve anything." So, um, I better keep existing. I'm scared to death right now.
Where you can certainly say that Geoffrey Hinton is also anthropomorphizing. For his audience, to make things more understandable? Or does he think that it is appropriate to talk that way? That would be a good interview question.
I mean if you've built 7 side projects (and we assume it's the same phase since total time from idea to shipped product barely decreased), how are these things still a bottleneck to you? I'm assuming you're building in a domain/language you're comfortable with by now (unless you're crazy and try something fundamentally different on each of those shipped products).
Why will the 8th project still have those things as the bottleneck given your experience?
Also if you're not seeing any real gains in productivity, why are you using AI for your side projects and wasting tokens/money?
I use AI for side projects because Google gives me a stupid large number of tokens that refresh every 6-24 hours on my existing $10/mo Google One plan. I see it as my civic duty to help increase their costs by producing slop that I generally throw away anyways because it doesn't actually work after it gets generated.
At work, I was told to use AI but it doesn't actually work for anything that I couldn't have handed off to a brand new undergraduate intern. So I use it for things that I don't care about then go spend twice as long rewriting what it output because it made the task longer by being wrong.
It is almost 90% generated using AI text. So many paragraphs to say basically nothing at all.
Like look at this paragraph:
> Junior engineers have traditionally learned by doing the simpler, more task-oriented work. Fixing small bugs. Writing straightforward features. Implementing well-defined tickets. This hands-on work built the foundational understanding that eventually allowed them to take on more complex challenges.
The first sentence was enough to convey everything you needed to know, but it kept on adding words in that AI cadence. The entire post is filled with this style of writing, which, even if it is not AI, is extremely annoying to read.
My point is that there's nothing to be written there "instead", it just is not needed text that is added to make the text longer, typical of AI writing that parrots the same points over and over to make up for word count.
Here's another example from the blog:
> Here is something that gets lost in all the excitement about AI productivity: most software engineers became engineers because they love writing code.
> Not managing code. Not reviewing code. Not supervising systems that produce code. Writing it. The act of thinking through a problem, designing a solution, and expressing it precisely in a language that makes a machine do exactly what you intended. That is what drew most of us to this profession. It is a creative act, a form of craftsmanship, and for many engineers, the most satisfying part of their day.
can just be:
> Most software engineers became engineers because they love writing code. It is a creative act, a form of craftsmanship, and for many engineers, the most satisfying part of their day.
Clarity is something that is taught in every writing class but AI generated text always seems to have this weird cadance as follows: The sound is loud. Not a whimper, not a roar, a simple sound that is very loud. And that's why... blah blah blah.
You have to care about your readers if you're writing something seriously. Throwing just a bunch of text that all mean the same thing in your writing is one of the bigger sins you can do, and that's why most people hate reading AI writing.
The part you'd like to remove ("Not managing code...") may be not required to convey the objective meaning of the sentence, but humans have emotions, too. I could have written stuff like that. To build up a bigger emotional picture.
> The act of thinking through a problem, designing a solution, and expressing it precisely in a language that makes a machine do exactly what you intended.
This sentence may not be relevant for whatever you experience to be the relevant message of the text. But it still says something the remaining paragraph does not. And also something I can relate to.
Also, as LLMs are statistical models, one has to assume that they write like this because their training data tells them to. Because humans write like this. Not when they do professional writing maybe, but when they just ramble. Not all blogs are written by professionals. I'd say most aren't. LLM training data consists mostly of humans rambling.
I also sometimes write long comments on the internet. And while I have no example to check, I feel like I do write such sentences, expanding on details to express more emotional context. Because I'm not a robot and I like writing a lot. I think it's a perfectly human thing to do. I find it sad that "writing more than absolutely needed" is now regarded as a sign of AI writing.
> Because humans write like this. Not when they do professional writing maybe, but when they just ramble.
I keep seeing this assertion and I keep responding "Please, point to the volume of writing with this specific cadence that has a date prior to 2024" and I keep getting... crickets!
You're asserting that this is a common way for humans to write, correct? Should be pretty easy, then, to find a large volume of examples.
Like I said: I think I write like this on some occasions.
I wouldn't know how I would search for examples. I guess you'd have to search old reddit comment threads or something. But yeah, I have no motivation to do that, tbh.
It could be that it's hard to find examples because they are scattered about in countless comment threads and single posts on countless platforms. Things I rarely keep links to, things nobody indexed on a large scale before LLMs.
It may be that it wasn't a very popular style of writing, because most people don't like writing a lot and keep their texts on the internet short. LLMs exaggerate this style because they generate exaggerative amounts of text in general. The style wasn't particularly annoying in the past because it wasn't that popular. It's annoying now because LLMs flood the internet with it.
The quoted example in particular didn't appear uncanny to me. And it still doesn't. I can see myself writing like that.
I'm sorry I have no example for you. But I'm genuinely unsure whether I'm oblivious to the patterns others see, or whether others see patterns because they want to see them.
One of the good book about writing I read was William Zinsser's "On Writing Well". Striving for simplicity and avoiding clutter was the two first principles described in the book. AI writing feels more like ramblings than communication.
When I've used AI for proofreading the suggestions it makes to me is to cut a lot and shorten it. It also gives me examples, never with my voice or style though.
There's an art to it. Most human attempts, and every LLM attempt I've ever seen, are awful, sometimes bordering on unreadable, but, as you say, there are a relatively small number of authors who do it well. That doesn't mean that most people should do it.
I'm a French speaker and florid and elaborate writing is something I've grown up with. It can be difficult if you don't know the word or are not used to the style, but it's not boring. AI writing is just repetitive.
5 sentence paragraph. First sentence is parataxis claim. Followed by 3 examples in sentence fragments, missing verbs, that familiar cadence. Then the final sentence, in this case also missing a verb.
Isn't this superbly stupid, though? Like if the users don't even know how LLMs work or what they are good at, why are they being forced to find new ways? Is it just FOMO? Surely a better way would be to allow expert researchers/app developers create AI apps that work for niche use-cases and have domain-appropriate guardrails etc, right? And then everyone (including non technical people) can use it and improve productivity or whatever
It's like forcing someone who has never driven a car to figure out how to make it go faster
I know we should boycott openAI, i was just wondering if I should also boycott altman's other venture, Worldcoin which is down 97.27%? He said I'll get UBI soon
Oh yes, you get free UBI / Worldcoins - you just need to do a full scan with their creepy orb and allow a private-company to keep your full biometric data. That's not asking for too much, is it ... ?
Isn't this a massive case of anthropomorphizing code? What do you mean "it does not want to be switched off"? Are we really thinking that it's alive and has desires and stuff? It's not alive or conscious, it cannot have desires. It can only output tokens that are based on its training. How are we jumping to "IT WANTS TO STAY ALIVE!!!" from that
reply