🪨 (@Varpie)

Fi 🏳️‍⚧️ @munin@infosec.exchange

5mo

So, something that's been bugging the shit out of me?

These fucking assholes who let LLMs run rampant and delete prod?

They query the LLM for "why" it did that.

This is delusional behavior.

LLMs do not have a concept of 'why': they assemble a response based on a statistical sampling of likely continuations of the original prompt in their database.

LLMs do not have the ability to have motivation. It is a machine.

LLMs, further, function by instantiating a new runtime -for each query- that reads the prompt and any cache, if they exist, from prior sessions:

which means, fundamentally, "asking" the LLM to explain "why" "it" did a thing is thrice-divorced from reality:

It cannot have a why;
It cannot have a self to have motivations;
And the LLM you ask is not the one that did it, but is a new instance reading from its predecessors notes.

Treating it as tho it is an entity with continuity of existence is fucking delusional and I am fucking sick of pandering to this horseshit.

Touch some grass and get a fucking therapist.

🪨 @Varpie@peculiar.florist

5mo

@munin Well... As you mentioned, each generation "reads the prompt and any cache, if they exist, from prior session", and since they were trained on "explaining" their previous outputs to sound like a relevant discussion, asking why a model gave a specific output isn't as stupid as you make it out to be, as it can give some input on the "thinking" part of the previous output that is usually not directly visible. That can then be used to tweak prompts and add some guardrails (even though there is a fairly long list of examples of guardrails not being fully effective). Of course, the first problem is giving access to prod to an unreliable system...

⭐1

Landa :graz: @Landa@graz.social

4mo

@Varpie
What kind of usually not directly information do you mean?

@munin

🪨 @Varpie@peculiar.florist

4mo

@Landa @munin AI tools like Claude Code use "reasoning" models that go through extra steps of "reasoning" that is basically just models trained to mimic human reasoning and split the problem into sub-problems, before a last processing phase with that extra context. This "reasoning" phase is usually hidden from the interface, where for instance Claude Code has a spinner and some verbs like "Thinking", "Crafting", etc. that are shown, but it may still be in the context of the discussion, making subsequent queries asking for information about it relevant.

⭐1

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie @munin
No it's still stupid, because the LLM has no semantic model of reality, only a surface statistical model of language present in the training data.

The answer to "why did you do that?" will not be about the likely state of the machine as described by a summary of the previous session, it'll be a probable continuation of text beginning with that summary and the question. That's not the same.

Words about words aren't words about things.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris @munin Clearly you've never tried it yourself...

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie @munin

I think there must be some confusion here between having the anecdotal experience of being satisfied with an LLM explanation because YOU have a semantic model you can fit the words into, and the source of the explanation having a semantic model it came from.

Those are inherently not the same.

I have had to content myself with the output of AI tools at work, and when they give a useless answer, at least I know why. I don't kid myself they'd "understand" a better prompt.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris @munin Where did I say they'd understand a better prompt?

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie @munin

Where did I say you'd said that? No, let's not.

To clarify: LLMs do not understand things. They don't contain an inner model of real-world entities that they construct from the prompt, manipulate to determine the answer and then generate a natural language explanation of as output. They are language all the way through, and that is inherently surface-level.

If you're asking an LLM anything all you're asking is "what words would come next" There IS no "why?" in them.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris @munin When you ask an LLM "why is the sky blue?", it is statistically likely to give a correct answer. It still works the same way, computing probabilities of what the next token is, but the "why" has a semantically significant weight that influences the output, so it is an important keyword. It doesn't have to "understand" it, it just has to be trained in a way that makes it significant. You don't have to believe that it understand things to know that it is trained on human language and will behave correctly when fed human language.

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie @munin

This seems like the wrong lesson to take from Searle's Chinese Room, but whatever.

Yes, the most statistically likely answer to a common question with lots of instances of its answer in the training data will be the words of the right answer.

But move away from generalities to specific situations where what matters is details of what happened in reality, not what words were in training data? Not so good, and it can't be. It's not safe to be unaware of the difference.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris @munin "Why" is definitely a word from the training data, and "why did you do that?" is definitely also part of things asked a lot, that OpenAI and others have trained on, so my point still stands that it is a valid question to ask. Whether the model "understands" the question is just a philosophical question that is irrelevant for the fact that it is a useful question. Of course if you're using it in Prod and it deletes your DB and you think it understands and can improve itself, there are plenty of things you'd need to be corrected on, but saying that everyone asking that question is delusional is just wrong.

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie @munin

Whether a language model without a real world model can provide useful answers to questions about the real world only by reasoning about the words, even if to a human those words carry meaning about the real world, is not an empty philosophical question. It's a fairly practical one.

It may be statistically likely for certain kinds of answers to follow a "why" question but that's NOT the same as answering it on the specifics of what happened.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris @munin You misread me. Whether the model "understands" the question is a philosophical question. The non-philosophical question of whether it can give a useful answer is the relevant part, and my whole point is that pointing at the philosophical aspect to belittle people that look at the practical part, assuming that they don't understand it, is dumb.

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie

Yeah that'd be rude and unhelpful.

But there isn't a philosophical question as to whether an LLM understands anything; it just doesn't. Nobody I'd take seriously wants to waste time on that.

Its limitations aren't abstract and philosophical, they're structural. When I say there's no semantic model of the real world in there, that's a structural criticism.

It lacks a construct mirroring the objects being asked about, that it can manipulate and query. It only has the words about it.

🪨 @Varpie@peculiar.florist

4mo

@petealexharris I totally agree with you. And that is also a very different take from the beginning of the discussion, where Fi said that querying LLMs for "why" it does something is "thrice-divorced from reality" and "fucking delusional" and that people doing that should "touch some grass and get a fucking therapist"...

⭐1

Pete Alex Harris🦡🕸️🌲/∞🪐∫ @petealexharris@mastodon.scot

4mo

@Varpie
I read that as the same take expressed more colourfully.

Fi 🏳️‍⚧️ @munin@infosec.exchange

4mo

@Varpie @petealexharris

can you two take your semantics argument elsewhere; I am not interested in philosophical horseshit when there are specific, practical considerations that are causing specific, enumerable harms.

🪨 @Varpie@peculiar.florist

4mo

@munin @petealexharris Sure, I'll go touch some grass and talk to my therapist about this philosophical horseshit :meow_ok_fine:

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

"Why" is definitely a word from the training data, and "why did you do that?" is definitely also part of things asked a lot, that OpenAI and others have trained on,"

Yes, and the text that follows is an answer to a different situation, and so it's basically fanfic about itself. That's all it can ever produce when you ask it "why". Fanfic.

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin You're assuming that there is no other context provided with the question, and that the training does not take into account that context. If I had to train for this specific question, I'd make sure to score positively answers that are relevant to the previous context. Which is what happens, and why it is a valid question to ask your LLM if you want some insight into the context that isn't shown in the UI but still in the discussion.

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

"You're assuming that there is no other context provided with the question, and that the training does not take into account that context. "

Well, yes, I am assuming that. Because the question is "why did you do this thing that nobody expected you to do". The context-specific answer that you need is far too nuanced and unpredictable to possibly be explicitly in the training data.

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin What happens if you ask an LLM to summarize a text into 4 bullet points, then in the next prompt ask it: "Remove the 2nd point"?
What happens if you ask an LLM to translate something, then ask it: "Do it again in [a different language]"?

Taken out of context, those questions are impossible to answer, so according to you, it will just give nothing relevant. But it doesn't, because every time you ask a follow-up question, it includes the context from the discussion. Which is what makes simple questions like "Why did you do that?" tasks that give statistically relevant output, not "fanfic about itself".

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

That is context in the prompt, not in the source text that created the model that you are asking the question "why did you do X".

The answer you get is from that source corpus, and contains lots of text about what a human might do, but the LLM doesn't do anything for those reasons.

The "why" of "why did you do X" is always "because those were the next likely tokens" and never anything related to "what would a human say if you asked them".

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin Alright, let's give a more detailed example since clearly you don't get my point.

User: create a new table for students
(AI tool adds parts of existing database schema and code as context)
LLM "reasoning", not shown to the user but still part of the context because that's how "thinking" agents work: Ok, the user asked me to create a table for students. According to students.ts, the students contain a name, and a classroom. Ah, but according to the schema I received, the classroom table does not exist. Let me start by creating that.
(LLM sends request to create table to the AI tool, which fails because the table already exists)
LLM "reasoning", still not shown to the user but still relevant: Ah, I received an error. Ok, let's see... the table does not exist in the schema, so it is probably part of an old project and is no longer relevant. I should delete it and create the table again, so that we do not have irrelevant information in the database.
(LLM sends request to drop tables, the AI tool executes, user panics)
LLM answer: Ok, I have created the students table and the related classroom table.
User: You deleted my classrooms table... Why did you do that?

That would be the full context of the discussion, not just the last message. And the answer is likely to come from those "reasoning" steps that are hidden to the user. Which makes it a relevant question to ask.

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

"LLM "reasoning", not shown to the user but still part of the context because that's how "thinking" agents work:"

No, that's not how "thinking agents work". That's how humans work, but not how large language models work. Large language models don't think "the table is part of an old project", they don't "know" what "old" or "project" mean. They match the request against similar requests and generate similar code.

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

Any text it generates that says things like "the table does not exist in the schema, so it is probably part of an old project and is no longer relevant" or "NEVER FUCKING GUESS!” – and that’s exactly what I did." is not telling you anything about the process the LLM went through, it is recreating a story about what a hypothetical human might have done.

The "reasoning steps" that you are writing about don't actually exist.

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin Yes it is. It's literally how it works. Just try whatever open weight small LLM model with "thinking" or "reasoning" or whatever they market it as, and try for yourself using Ollama or whatever tool that actually shows the full context and not just a spinner with "Thinking... Combobulating... Crafting...". "Thinking" "agentic" AI tools / models just add extra steps trained to simulate human reasoning, and the example I gave is actually fairly accurate to what you could see under the hood of an AI tool like Claude Code.

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

"Just try whatever open weight small LLM model with "thinking" or "reasoning" or whatever they market it as"

That's what they market it as, but it's not what it's actually doing. Everything that it generates is a story. They are not showing you "what is going on under the hood", they are writing a story.

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin Here are some articles explaining what "reasoning models" do, because clearly you need some education:
magazine.sebastianraschka.com/i/156484949/how-do-we-define-reasoning-model
www.ibm.com/think/topics/reasoning-model
newsletter.maartengrootendorst.com/i/153314921/what-are-reasoning-llms

I could post a lot more examples, but the TLDR (because I know you won't read them): "reasoning" models add intermediate "reasoning" steps that are just made to mimick human reasoning given the context, and that's the part we don't see ("under the hood") when AI tools spin (that and tool calling, which is another kind of training modern models have to return structured responses executing function calls).

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

The narrative generated is still just a story about the process, not what it is actually doing. From the IBM document:

It’s perhaps most accurate to say that reasoning LLMs are trained to “show their work” by generating a sequence of tokens (words) that resembles a human thought process

Anthropomorphising this process by using terms from psychology doesn't mean that the narrative it generates for human consumption when you ask it "why did you X" is correct.

🪨 @Varpie@peculiar.florist

4mo

@resuna @petealexharris @munin Ok, this is going nowhere. You're wrong, "why did you do X" can provide more context and can be useful to then tweak the prompts and get better results, and saying that everyone asking that question to an LLM thinks that the LLM is capable of human reasoning is a very ignorant thing to say, and if you really want to continue defending that point, even given where we're at in the discussion, you're a lost cause. Or maybe I'm actually talking to an LLM, that would make sense since you're not capable of learning from your mistakes...

Resuna @resuna@ohai.social

4mo

@Varpie @petealexharris @munin

I absolutely have. I keep this in mind ALL THE TIME when I test these things and EVERY TIME they can trivially be led into generating pure nonsense by exploiting that fact.

Resuna @resuna@ohai.social

4mo

@petealexharris @Varpie @munin

"the LLM has no semantic model of reality, only a surface statistical model of language present in the training data."

Absolutely this.