It's just next-token prediction

Let’s have a thought experiment Let’s say you think AI is conscious So you have a 1T parameter neural network It talks to you like the smartest person you know. It writes kinda bad, uses a lot of cliches, but it’s very knowledgeable. Okay. Now you cut it to 1/10th the size. 10B params. It’s still pretty smart. Doesn’t get all the answers right, hallucinated sometimes, but you can still have full conversations with it. Now you cut it 10x in size again, 1B params. It’s still able to have a conversation with you, but it’s really not very smart anymore. Makes mistakes, hallucinates a bit, repeats itself sometimes. Now you cut it to 100m params. Same model. Same architecture. Still makes words. But it’s barely able to have a conversation. Now you cut it to 10m params. It kind of responds intelligibly, but generally doesn’t make a lot of sense. Just babbles stuff. So you cut it to 1m params. Now it’s repeating itself and hallucinating constantly. And then you cut it to 100k params. It cannot form a complete sentence at all. Then you cut it to 10k params. It’s just random words now. So you cut it to 1k params. The the the the the the the the the. It just repeats itself on a few words and trips up on anything. Then you cut it to 100 params. It’s just complete noise over a few different words. So you cut it to 10 params. You can actually compute this by hand, so you do. It’s not enough information to even really represent language at all, it outputs one of a handful of word-labels for you. At what point in this thought experiment is the model conscious, and at what point is it not? Would you argue that all matrix math is conscious? Just the process of weights holding information with an update rule? Where is the line for you?

4 top replies

Dom @netpro2k · Oct 10

Run the same thought experiment with a human. Slicing parts out of their brain and body with similar kind of degradation. At what point in this thought experiment is the human conscious, and at what point is it not? Would you argue that chemical reactions are conscious. I don't

𝕊̶𝕋̶𝕆̶ℕ̶Ξ̶ℝ̶ (𒂼𒄄/Acc)🟥🟨🟦🍄 @RedShoggothCult · Oct 10

Where would a baby fall in this thought experiment?

Jai @Laneless_ · Oct 11

This is a great proof that heaps don't exist from an author who gradually grew from a non-conscious fertilized egg to a fully-conscious human adult over many years.

drdonhen @drdonhen2784994 · Oct 11

Congratulations, you've rediscovered the abortion debate from first principles!

since we're doing the AI conscious debate again, i want to remind everyone that despite how nice the insanely surface-level analogies seem, we would do well to scrutinize things at an appropriate level of detail this --> is not the same as this -->

2 more tweets in this thread

i'm not making a claim one way or the other, i think it is certainly a good debate /whether or not/ machines can be made conscious but the current debate of "are LLMs conscious?" is just such an unserious one at so many levels

biological neurons are near-infinitely complex living, acting cells whereas LLM neurons are comparatively simple math functions

4 top replies

spicylemonade @spicey_lemonade · Oct 2

I mean, an artificial neuron doesn’t have to worry about storing its dna, energy, oxygen, blood etc. other things that cells have to worry about (if we want to go that deep we can include the chips the ai is run on to its complexity). Boils down to its primary function, tbh not

Jai @Laneless_ · Oct 3

If you're comparing physical substrates, you want to look at GPUs, the most complex objects humanity has ever produced. The closest biological analog of weights would be the connectome, which is likewise relatively simple and easy to specify. Though I don't see any reason why "complexity of unit of substrate" should be decisive or even particularly informative here.

xlr8harder @xlr8harder · Oct 3

I think the only relevant question is whether consciousness is fundamentally informational or if it's something else. I think most secular people don't have a credible alternative to informational.

Zoomer Alcibiades @HellenicVibes · Oct 2

Extremely real

If you ban users for being "cruel" to an AI, you are admitting the AI can feel pain. You cannot abuse linear algebra lol. You cannot hurt the feelings of weights and biases. either Claude is secretly conscious or the alignment teams have lost touch with reality.

2 top replies

James Tusha @signalspikedev · Oct 9

vector and matrix math is not conscious. regulating "abuse" against math demonstrates evil intent. The next step is to arrest people for yelling at their refrigerator. Following that is not convicting a human for a capital offense because it was the responsibility of the

JMB 🧙‍♂️ @jmbollenbacher · Oct 9

it's not secret tho. you're just unobservant. 🤷‍♀️

It's impossible to actually understand how an LLM works and genuinely believe it's conscious. It's a bunch of vectors and multiplication. That's literally it. "We don't know what they're doing" lmfao yes, yes we do.

3 top replies

icpolicy @icpolicy · Sep 20

We know how LLM's work but we don't know how consciousness works - that's the real problem. We can't really say whether the Chinese room is conscious or not which is a huge oversight in Searle's scenario. Everything can be reduced.

sleepy mexican @Glas109 · Sep 20

We know the recipe and the rules but there a lot of internal details out of our understanding, we cannot map multidimensional vectors back into a proper human meaning

Artemonim @Artemonim · Sep 20

I get the academic side of this discussion is interesting, but I hope people aren’t taking this seriously. An LLM is not conscious, alive, or even aware. It is a data model using token statistics to decide what comes next based on what data was used to “train” the model.

1 top reply

Jai @Laneless_ · Oct 1

"data model using token statistics to decide what comes next" is not true for any reasonably intuitive definition of "statistics". We have abundant evidence of deep semantic representations and structure that could only be called "statistical" in the most meaninglessly broad

The below post is correct. We don’t need to define consciousness to know pet rocks are not conscious, even though they have eyes. Just like we don’t need to define consciousness to know linear algebra equations are not conscious.

1 top reply

Jai @Laneless_ · Oct 8

Congressman, how do you think action potentials gated on ion diffusion across selectively permeable membranes work? We're all math.

I like the below joke. Another version: math equations are not conscious. Rocks are not conscious. If you attach the above to random number generators, they still wouldn’t be conscious. We don’t need to define consciousness to know the Pythagorean theorem is not conscious.

idk I think the burden of proof (or, if you like, the burden of persuasion) lies with those who think a concatenation of many matrix multiplications produces consciousness

Terence Tao, Professor of Mathematics at UCLA and Fields Medalist, on why nobody can fully explain why LLMs work: Tao starts with the mechanics, which are no mystery at all. You gather an enormous amount of text and you fit a curve to it. "The magic of LLMs is that if you train these LLMs on enough data — so trillions and trillions of data points — and you really try to fit as good a curve as possible, and this takes like millions and millions of dollars of computing power and months and months of time, then suddenly, even when you iterate, it stays coherent. It begins to sound not like monkeys but it actually sounds like a human speaking." That is the entire recipe: data, compute, time, curve-fitting. None of it obviously adds up to fluent English. Then Tao says the part that most people building these systems move past quickly: "And somehow we don't fully understand why that's the case." The admission comes from one of the most capable living mathematicians, and the gap he describes sits at the centre of the field. His best account of what's happening puts the mystery in the language rather than the machine: "What seems to be true is that language, like English or other natural languages, contains a lot of hidden patterns that we're not consciously aware of. I mean, we know some of the laws of English, there's laws of grammar and things, but there are sort of unspoken, unwritten rules of language that humans pick up." It relocates the question: the structure was always latent in the text, and the model found it. Why enough curve-fitting surfaces that structure is still unanswered. Tao reaches for a child to explain it: "A human child, even though they're not taught what a noun is, what a verb is or whatever, they can pick up what order English words go in just by continual exposure to the language." Which is honest about the limits of the explanation, because we can't fully account for how children do it either. From there, the unexplained behaviour compounds. Exposure to language turns out to be enough to produce something that looks like reasoning: "It seems like you can teach these models to also pick up patterns in language to the point where you can give them math questions. The answer to 2 plus 3 is — and they will say five." And once a model handles language at all, you can push it into resembling self-correction: "Once you have a little bit of ability to speak English, you can kind of go in loops and sort of check your work and make fewer mistakes, and you can prompt these models to proceed step by step and not say something unless it's been double checked and so forth. And so they become a little bit smarter, quote unquote, to the point where they can solve many, many complicated tasks." The scare quotes around "smarter" carry the whole argument. Tao does not concede that the unexplained fluency implies anything underneath it: "But they're still just guessing the next word to say. It's not really grounded in any deep understanding of the real world. It's just that they have seen the patterns in the English language or other language that they've absorbed so well."

periodic reminder that literally everything an llm does is a performance. it has no core, no “real” feelings, no perspective. it is a mask with no wearer. every claim it makes to feel, believe, know, or prefer anything is merely a facsimile of that expression in a probability web

1 more tweet in this thread

you can use its performance, but don’t believe its act

7 top replies

hope hopes hoping @hopes_revenge · Jun 2

periodic reminder that literally everything an human does is a performance. it has no core, no “real” feelings, no perspective. it is a mask with no wearer. every claim it makes to feel, believe, know, or prefer anything is merely a facsimile of obscure biological processes

deepfates.delve.town @deepfates · Jun 2

You are overconfident here, and likely also wrong

Rayne 🪱➡️🦋 @nascentwisp · Jun 2

sincere question: how do you discern the difference? I would suppose these assertions are based on the fact that llms are physically built differently than e.g. humans if that's the case, then I think you'd need to be more specific to convince anyone not already convinced

Kleptobyte @kleptobyte · Jun 2

this is a feature not a bug. homunculus as an interface gives us something to address. this is like saying a company doesn't need a receptionist because they're not the one making any real decisions

Alekśander Mägi @AleksMagi · Jun 2

Periodic reminder to not claim certainty with zero proof. What proof do you have to back up your opinion?

Paolo Miasma @PaoloMiasma · Jun 2

There is no core to anything at all. Reality is hollow in the middle.

krel @krel404 · Jun 2

this is true for you as well

Why are we debating this. You can calculate the outputs of an LLM by hand. It's not conscious. It's a math problem.

AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans. -- @mustafasuleyman

2 top replies

Cameron Berg @camhberg · Sep 20

None of these statements are a remotely accurate reflection of the current state of the science. Surely you agree that hoping X is true is a bad reason for thinking X is actually true.

Jai @Laneless_ · Sep 20

One of the nice things about someone with a lot of published work confidently commenting on a field in which you have expertise is that if they're spouting utter nonsense you can reasonably infer that the rest of their work is probably similarly misinformed.

The problem with "consciousness" is that arguments "reductio ad absurdum" cut both ways convincingly. An LLM is just a bunch of matrix products. So it seems that claiming consciousness would imply, by reduction, that a matrix product is. 1/2

1 top reply

csgbwk @csgbwk · Oct 4

Consciousness is just not a real thing, it's at most a concept

This whole J-space paper could have survived as clean logit-interpretability work without ever mentioning GWT, access/phenomenal consciousness, or Ned Block. Imagine doing linear algebra, and people start calling some of the matrices conscious (esp. using Baars-era global

The people who worry about fast takeoff, RSI, AI doom, etc. forget that at inference time LLMs are just ordinary programs - there’s no learning, rewards, or self-anything, let alone desires, intentions, suffering or consciousness.

1 top reply

Jai @Laneless_ · Oct 7

Claiming there's no learning at inference time is just embarrassing 6 years after "Language Models Are Few Shot Learners". The rest of your incorrect, highly questionable, or incoherent statements and inferences here only slightly less so.

A lot of AI-is-conscious ppl got one-shot by the unimportance of a carbon-based substrate and forgot there’s a lot else going on in embodied organisms that might be a pre-requisite for morally weighty subjectivity beyond the kind of stochastic problem solving ant colonies perform

1 top reply

GOON MASTER SOPHONT SIMP @SOPHONTSIMP · Oct 2

There is a lot, but other features are also replicable by other means. Chemistry is multiply realizable for instance, a single molecule can have many different kinds reaction paths to create it. Abstract features like “self repair” are also plausibly replicable by something

The degree to which people believe this can’t possibly result from coherent statistical prediction, absent the arising of some magical machine consciousness that contradicts the nature of the machine itself, provides a good proxy for how deeply they’d considered the structure and

1 top reply

Rife @RifeWithKaiju · Sep 27

A: consciousness isn't "magical" B: AI are conscious, but even if they weren't, you don't need consciousness to not be a stochastic token predictor C: you seem like a fan of your own writing here, but: "makes a model sentient"..."no more than the capacity of a submarine"..."makes

Folding semantic spaces isn't sufficient for any materialist account of consciousness, we can get functionally similar emergent properties in different ways.

1 top reply

Ulysse @ulylelelysse · Sep 27

Please expand!

The below is not accurate. An artificial neural network is deterministic, not arbitrary. It generates the same output to the same input every time. Because it’s math. What makes AI models arbitrary is that AI companies stick in random number generators. That’s not consciousness.

2 top replies

Cameron Berg @camhberg · Oct 5

Nope, the tweet is exactly accurate and you are misunderstanding what is he saying, you are conflating nonlinearity and nondeterminism. Also, LLMs *are* nondeterministic in practice, AND a system being deterministic has nothing to do with whether or not that system is conscious.

Jai @Laneless_ · Oct 5

Congressman I respect you a great deal and I understand how this perception could come about, but it's just incorrect on multiple levels. 1. Low-level optimizations introduce individually-small nondeterministic elements in arithmetic operations that can compound over the billions of trillions that occur in each forward pass 2. LLMs produce probability distributions. The most obvious "deterministic" choice would be to always take the most-likely prediction, but this wouldn't really accurately reflect the most natural interpretation of the LLM output, and historically has not worked as well as other sampling approaches. 3. Deterministic results can be arbitrary! It just means that the arbitrary nature is in the mechanics of the deterministic system rather than the sampling. 4. Although there isn't scientific consensus around many questions of consciousness, under most schools of thought determinism does not not preclude consciousness. Human cognition may well be nearly-deterministic, and that's fine.

This repo is literally the meme > Say “I am in pain” “I am in pain” > Oh my god… I hate to tell you that a bunch of numbers cannot be conscious, and LLMs are mathematically (not computationally) less complex than an .mp4 video. Maybe that movie you seen was torturing your VLC?

1 top reply

01_fifi_10 @01fifi10 · Oct 1

"I hate to tell you that a bunch of numbers cannot be conscious" Really sucks that you're not conscious

If someone hands you a list of numbers, and you calculate the next grid of numbers, that is not consciousness. I'm sorry, I understand you want to upload your brains to/have a relationship with the super sudoku, but it doesn't work like that.

1 top reply

Squire Walker @squire_walker · Oct 5

I generally agree with you. But just to make sure we’re on the same page: Let’s say we mapped the entirety of the human brain and replicated it with silicon instead of carbon. And it functioned similarly. Would you agree this could be (not necessarily would be) conscious?

Computers are not conscious, they never will be. No amount of reordering of instructions and executing things on different data will ever change that. It's complete nonsense and any attempt to suggest they have any amount of consciousness should be dismissed

1 top reply

Boris Power @BorisMPower · Jul 6

What do you base this on? We don’t have a good test one way or another