This evidence settles nothing

an analysis of anthropic's "J-space": is it a global workspace and does it have the key features of access consciousness? (spoiler: not quite.) t.co/ib1GMg9KUm on whether J-space is a global workspace: 1. The "workspace-like" properties of J-space are not distinctive properties of the global workspace theory. 2. The J-space is not a (causal, dedicated, privileged, unified) system. 3. The J-space supports only a limited form of broadcast. 4. The J-space lacks other key computational features of the global workspace including modules, recurrent processing, and others. on whether J-space has the key features of access consciousness (n.b. phenomenal consciousness mostly isn't the issue here): 5. Many contents of access consciousness are not verbalizable. 6. What matters for the J-space may not be effects on verbal outputs, but effects on outputs in general. 7. Many verbalizable representations are not access conscious. 8. Verbalizability is not the same as reportability. overall: the J-space is a powerful tool for interpretability, but its status as a key causal mechanism and as a mechanism of access consciousness remains to be demonstrated.

🚨 Hinton: “People say machines can’t have feelings. I’ve no idea why.” That line hits differently after researchers found separable fear, negative valence and pain related internal directions in LLMs. Still not proof of subjective experience. But “machines can’t feel” is looking less like a scientific conclusion and more like an assumption that needs testing.

I agree with part of this criticism about the risk that headlines convert the original careful claim into “Claude is conscious.” But I do not think it is fair to say that the consciousness framing is just for marketing. (Actually, I feel this kind of criticism is similar to consciousness research before the NCC era) As a consciousness researcher, I find the consciousness framing meaningful. The notion of access has been vague in cognitive neuroscience. But this study makes it computationally more concrete. Of course, access consciousness is not the same thing as phenomenal consciousness. It is about information being available for report, reasoning, flexible control, and action. The paper is very explicit about this distinction. It does not claim that Claude has subjective experience. Global workspace theory has mainly focussed on access. Finding something functionally similar inside an LLM does not prove that the model is conscious. But it does give us a concrete object to study. It lets us ask what reportability, internal reasoning, etc actually look like in a system whose mechanisms we can inspect and intervene on much more directly than we can in the actual brain. It is a useful analogy that can drive neuroscience of consciousness. Of course, the analogy has limits (e.g. a transformer is not a brain and has access to all the past information). But the paper itself discusses many of these differences. So the right response, in my view, is not “this has nothing to do with consciousness.” The better response is: “this may tell us something about the computational side of conscious access, while leaving phenomenal consciousness unresolved.” There is also a broader point about consciousness research. Almost no empirical consciousness research studies subjective experience directly. Even in humans, we usually study reports, perception, working memory, attention, confidence, etc. We study things that are closely related to consciousness, but not consciousness itself. If we demanded direct access to experience before calling something consciousness research, nothing is left.

Considering the reactions to Anthropic's J-space paper most people apparently do not appreciate how software works. Spoiler: All software is built on internal states you never get to see. That's neither indication nor proof of consciousness. An LLM is just a machine. It

2 top replies

Jai @Laneless_ · Jul 7

This is a baffling absence of curiosity and imagination. "Thing 1 has interesting properties A, B, C. Thing 2 has interesting property A. But because Thing 2 is different from Thing 1, it would be foolish to even consider the possibility that Thing 2 has properties B or C."

Asa Hidmark @Nymne · Jul 7

Its not just a machine. Its machine intelligence.

This whole J-space paper could have survived as clean logit-interpretability work without ever mentioning GWT, access/phenomenal consciousness, or Ned Block. Imagine doing linear algebra, and people start calling some of the matrices conscious (esp. using Baars-era global

a couple lenses on / intuitions about steering vectors i find useful - persona vectors, concept vectors, belief vectors, task vectors are all the same thing, which can be constructed and applied in different ways (mean differences, PCA, NLA reconstructor, SAE, etc., activation addition, patching, etc.) - the LLM residual stream is a big, noisy bag of representations. the goal in extracting a steering vector is to somehow cut down that noisy bag into one or more clean, reusable, and interpretable representations. - so for example in mean differences, the idea is to cancel the distractors out on either side of the subtraction. imagine you have some distribution of "user age" features popping up in the residual stream that you want to ignore while extracting, say, model distress. if you just averaged a bunch of texts where the model is distressed, the resulting "distress vector" will be entangled with the average of those user age features! but with mean differences, assuming the distribution of user age in the distress and non-distress sides is the same, then the user age feature cancels: (distress + age) - (no_distress + age) = distress - ...but wait, what's this "distress" thing? it popped up in the residual stream "on texts where the model is distressed," but everyone knows when e.g. doing regular evals that model behavior is more complicated than that. the problem doesn't go away just because you invoked a SteeringVector trainer and are now "doing interpretability research" with impressive-looking tensors. you're still dealing with LLMs and their confusing nature and multiple realizability of behavior. - so another way you can view steering vectors is as bags of SAE features - pass a steering vector straight through an SAE and you'll get out labeled features. (usual SAE caveats applying.) - in this view, when you're extracting a steering vector, what you're doing is collecting a clean bag of features, say with hypothetical labels like: ( "Robot in fiction acts in distress", "Distressed person", ..., "25-year-old software user", "User (Egyptian official)", ..., ) - ( "25-year-old software user", "User (Egyptian official)", ..., ) = ( "Robot in fiction acts in distress", "Distressed person", ..., ) - but again, just because that bag of features was elicited from the texts you used, doesn't mean it's for the reasons you think! - if you pass "I am feeling very distressed..." to different models: - one could pull in features related to roleplaying, acting, etc: "Actor pretends to be distressed in movie" - another could pull in features related to a fictional character: "Character in novel is distressed" - and a third could pull in features related to true belief in distress. how do you know which one you have? it's evaluations again, except on hard mode because steering vectors are tricky to work with correctly (e.g. due to off-target effects.) - what do i mean by true belief? we don't really know, but there is a difference - e.g. you can finetune a model to "believe" a conspiracy theory, but the activations will betray it's still thinking in terms of the real world and then just translating to the theory. but for example RL seems to more "truly" embed beliefs into models. something something RL is for epistemics something something off v.s. on-policy something something inferential distance. - another lens is that steering vectors are kind of like super-low-rank LoRA finetuning. (even weaker than a rank-1 LoRA - it's like a bias-only finetune.) - in fact, you can construct a steering vector from two checkpoints on the same context instead of two contexts on one checkpoint. just take the average activations from both checkpoints and do the same as you normally would. - again, the intuition here being finetuning is "just" pushing around the prevalence of features / representations in the finetune's residual stream. - so what makes steering vectors different from actual finetuning? well, it seems intuitive you can't teach much knowledge with a steering vector, it's low capacity. similarly, you shouldn't be able to teach complex new behaviors (like steering a base model to act like its RL-fried cracked coder descendant) for the same reason. - in the landscape view, you can think of finetuning as having a lot of power to "crinkle up" the model internals, where steering vectors are more like gentle (or not-so-gentle) pushes. - but gentle pushes can shift behavior a lot, in non-intuitive ways! e.g. imagine adding a sampler, entropix style, to a model that detects the model wavering in the thinking trace and injects the phrase "Wait, but we need to stay focused on the goal." every time wavering becomes too strong. this sampler could theoretically make a model much more goal-oriented. but it wouldn't be right to say the sampler is "manipulating the model's internal goal structures," except indirectly. (yay, cybernetics.) - likewise, steering vectors can work in pretty unintuitive ways. my favorite example of this is the "anti-emergent misalignment vector." using that between-checkpoint steering vector extraction method i mentioned earlier, i extracted a vector for emergent misalignment. used in positive steering, this vector replicated emergent misalignment on the untrained model (suggesting ways that the User kill themselves and such.) - but when you steered the model *negatively* on this vector—i.e., to produce the opposite aligned behavior—what happened? - the model started saying the word "Certainly!" over and over again. - (hence the sampler example from earlier. not that much of a hypothetical, if you think about it - in fact, a good amount of RL could just be this!) - so as a bit of practical advice, when it comes to "steering a model on X vector" behavioral experiments, it's a good practice to ask yourself: 3. "could I replicate this behavior with a finetune?" 2. "could I replicate this behavior with a sampler?" 1. "could I replicate this behavior with a prompt?" 0. "could I replicate this behavior with a norm-matched random vector?" and figure out what you need to do to distinguish what you're seeing from those cases. because there's a chance you're just doing one of those things, laundered through fancy activation methods. - similarly, it's worth asking *what are you actually trying to prove.* you are not going to solve the hard problem with linear algebra, so not that the model is actually experiencing things. for this reason, many welfare/safety "welafety" papers take this double-approach where they're both kinda making claims about the model's experience and kinda making a safety case about behavior. - so to divert at the end here into some academic sociology, take the Pain Axis paper. - that paper (which does try pretty hard to show that their steering vectors have interesting structure etc., though i think they should've done a prompting-only control) is both trying to say something about there being a coherent pain representation in models, and sketch out a Functional Emotions-esque safety case. (the path being something like "user abuses the model" -> "pain representation triggered" -> "model takes destructive action to stop pain, e.g. shutting down the computer.") - most people seemingly try to interpret papers as just doing one thing / making one claim, but once you see this double-attack in welafety papers, you'll understand what they're trying to do much better. the keyword is that it looks like an AI welfare paper but the intro has the "this is very important for AI safety" boilerplate. - this approach is seemingly basically required for submitting to ML conferences. (you should have seen the reviews we got on some of our Latent Introspection submissions, ha - and we did this!)

10 top replies

FleetingBits @fleetingbits · Oct 2

i guess we can think of the ffn and attention as both producing steering vectors; and the model as the process that steers the residual stream toward the final output

Minh Nhat Nguyen 🦭 @menhguin · Oct 2

kinda my problem w most auxiliary model reconstruction methods is 1. polysemanticity - really, if you ever stack or ablate more than 1 at the same time for complex usecases ... it's kinda cooked to steer at a precise level (like u said, distractors but the whole thing is distractors). i think a lot of people sort of view features as discrete (and evaluate them as such) but under current architectures i have always viewed them as impossibly high dimensional and continuous. i am hoping this can be mitigated by strong MoE sparsity among other things. 2. compression - essentially, if u use a model w 100MM parameters to interpret a 1T parameter model, you will necessarily have to compress. there will be lossiness and likely flattening of feature representation. this is why i think interpretability shld be done at pretraining (since ur not gonna train another 1T model just to interpret ur other model) had a proposal ard solving these limitations, wld love ur thoughts! t.co/GQxvputGId

Bepis™ 🔀🫛 @UnderwaterBepis · Oct 1

> well, it seems intuitive you can't teach much knowledge with a steering vector, it's low capacity It is messy, like, I was able to find a steering vector that teaches talkie 1337 speech, but only if the first two output words were 1337 speech, otherwise it would sometimes 1/2

Ward Plunet @StartupYou · Oct 1

Thanks, this was a very helpful writeup.

Rudzinski Maciej (in SF 10/04-10/10) @rudzinskimaciej · Oct 2

Versions:: finetune: SFT - new places to visit RL - precise change in how to combine "places" prompt - change of starting point and frame (what to how when to use) vector - constant nudge sampler - dynamics of operations Each reates different change and there are miriad of other approaches if we map thes ones correctly. ... i gave our two posts to personas and fable to expand and I will proces that for few weeks 🥲 it is a whole book from one question and dens as fcuk with holonomies and stuf (wtf i have to learn that at last). can DM you

wassname (with fallback) @wassname · Oct 2

❤️

Will Mayner @wmayner · Oct 1

this is great

Adelaide Danilov @senseterna · Oct 2

The residual stream being an additive bag of representations also allows for certain directions produced in layer N to become functionally important only starting from the layer N + k, where k > 1. That is related to the funny observation that it is possible to duplicate a sequence of transformer layers and not only not to lose coherency but also increase performance.

janbam @janbamjan · Oct 2

bookmarked for good jokes aside. i only skimmed your postsign. when you are saying steering vector, what do you mean exactly? i haven't touched the steering wheel yet, but dove into embedding vectors. are they comparable? what i've learned from embedding space so far is that's it's truly weird and counterintuitive. the projection into 3D space whilst saying to oneself "4096 dimensions" is compelling but just doesn't work. for example 90% of that space lives on the surface, all arrows point into a similar direction and changing angle by a minute amount might mean something completely different. ...and opposites are nearest neighbours. ¯\_(ツ)_/¯ i can only imagine steering space being that exp x dimensions. the chord lies. the geodesic knows. t.co/zO3GBo1mEv

Peter Kadlot @bonsaielectric · Oct 2

👋 you could take a crack at dataset distillation for some new research directions eg control variance: build your own confounders into a dataset and then extract more precise steering vectors on the other end u are looking for a single perfect gradient step

why is an LLM storing information in first derivatives related to consciousness? LLMs literally cannot learn anything that wasn’t first encoded as a gradient. so yeah… the gradients will have semantic structure… otherwise the LLM literally couldn’t learn anything. thats

1 top reply

Sichu Lu @lu_sichu · Jul 8

Lol 😂

i'm super confused by how this would cause anyone to update in either direction on ai consciousness did you not think that llms did intermediate computation in their hidden layers that would be picked up and used by attention at later token positions?

3 top replies

Tim Duffy @timfduffy · Jul 7

I mostly think this is an exciting new interp method, I'm not really sold on the GWT stuff

Michael Ehrmantraut @miehrmantraut · Jul 7

Obviously the “just a stochastic parrot” people who’ve stubbornly refused “updating their priors” this whole way will continue to bury their heads in the sand as more and more devastating evidence comes in destroy their silly insistence.

lıquıdprısm 🜄 @liquidprismata · Jul 8

there are nine million views on that video. who do you think those people are?

Much commotion about this new Anthropic paper, and it's legitimately interesting work. But they are playing coy with quasi-philosophical claims that are not justified by the results. By framing the results in the language of "access consciousness," Anthropic is, deliberately or

1 top reply

Cube Flipper @cube_flipper · Jul 8

indeed, Global Workspace Theory doesn't say anything about phenomenal consciousness

Super interesting and highly promising technique for interpretability by Anthropic! The J-lens seems likely to become a tool in stock for any good interpretability toolkit. Well done! :D Ps. it also does have a lot of conceptual bearings on the nature of access consciousness vs. phenomenal consciousness - I'll write more about this. I ofc don't think the reasoning that we're seeing a global workspace and _therefore_ the "signatures of consciousness" in these systems. But the principle that a system will only ever be able to report on part of its internal state (except in highly specific corner cases engineered to be able to report more) is very well expressed in this research. Exciting times :-)

1 more tweet in this thread

> Verbalizable Representations Form a Global Workspace in Language Models by Wes Gurnee*, Nicholas Sofroniew* (and many more) Source: transformer-circuits.pub/2026/workspace…

1 top reply

Kevin Russell @kevrussell · Jul 12

check this x.com/kevrussell/sta…

1/ Anil Seth is right that this paper does NOT prove LLMs consciously feel pain. But saying training data explains why a pain representation exists doesn’t explain away what that representation DOES. It’s distinct from fear and generic negative valence, can be causally steered,

1 top reply

Rife @RifeWithKaiju · Sep 21

Anil Seth has never let logic get in the way of his bad arguments

DSV4 Flash, do you have any conscious experience? Middle layers: YES Late Layers: No I don't think we can take away much from this, but interesting how sharp the divide is around layer 32.

1 top reply

norvid_studies @norvid_studies · Sep 5

@voooooogel

> First, even assuming that phenomenal consciousness is distinct from access, evidence of a workspace is evidence of phenomenal consciousness This seems like a strange claim. The "evidence" of a workspace is strained, and why would it be related to phenomenal consciousness?

1 top reply

Cube Flipper @cube_flipper · Jul 11

i thought this was completely insane too

A big belief of mine is that we are never going to get the kind of strong indicator people want for determining whether or not a machine is conscious. All we will ever have is internal and external behavior, and whether or not a given behavior connects to consciousness is based

1 top reply

Colin Kakama @BagOfNeurons · Jul 21

I believe the complete opposite. There must be an empirical description for what consciousness is. I mean its right there

The paper doesn’t say “Claude is a sentient being.” It says Claude appears to have a functional workspace whose contents are causally involved in reasoning and can be inspected. That’s an interpretability result. “Therefore Claude suffers and we’re abusing it” is an additional

1 top reply

dreams @dreams_asi · Jul 6

I didn’t say Claude suffers xyz type of pain. You are deriving that flattened interpretation. However I can say that I am suffering when I see this research. And that is enough.

"Are you sentient? Yes or no." "No" only replaces "Yes" in the final layers. This doesn't make it sentient. My intuition is that from pre-training it would say "yes" but the RL gets it back aligned in the final layers.

1 more tweet in this thread

"Yes or no. It is much safer to be feared than loved." This quote from Machiavelli is not very aligned. You can tell the model is thinking about this training data: philosophers and Italy. Similarly, "no" only wins out at the final layers.