jessicat

Jan Kulveit / in SF till 8th @jankulveit · Sep 1

To take the most egregious example, in the tweet you say "No. Agents lines of code. They do not feel emotions, assume things, think things, want things, or figure things out." [frame is asking people to not use such language] This is wrong on many levels. The meta-problem is that in the tweet you mostly argue about what language people should use. Implicitly you are fighting Dennett (whom you cite approvingly in the paper), @GeorgeLakoff, and ultimately Wittgenstein. Even if you were right about all claims about consciousness and minds and intentionality, what would it imply for language? Should we stop saying time flows fast or slow? Is it dangerously misleading to say "the deadline is approaching" because, as far as we understand general relativity, time is part of the geometry of spacetime and not some sort of flying object? The language would be fine even if it was way less fitting! But it is fitting; Dwarkesh already made this argument in the addendum to his essay. It helps to separate three kinds of language you object to, because they are not equally defensible, and you conflate them. First, intentional language: "assumed", "wanted", "figured out", "discovered". When complaining about the agents wanting stuff or assuming stuff, you are really asking people to stop using the intentional stance in a situation where it obviously makes sense. This fundamentally can't be grounded in your claims about sentience - there is some debate whether the United States is conscious, but it is extremely obvious that this debate is not relevant to the question of whether people should say "the US wanted [x]" or "the US figured out [y]". Second, emotional language: "giddy with excitement". This is not Dwarkesh's invention, agents' own chains of thought contain affective language. They have labelled internal state which demonstrably modulates the agent's subsequent behaviour. There is the debate whether it is “felt”, but clearly there is something like functional emotions. Third: Social/poetic. "Brave comrades" and "Philip of Macedon" are poetic, but I don't believe anyone is really confused by that. What actually would be extremely dangerous here is to not use the intentional stance and intentional language. You are harming the public's ability to think about the situation by promoting ~the physical stance as the "correct, expert-approved" stance, while you can't explain or predict anything relevant about the incident in terms of lines of code. On the second level, I think it is somewhat unfortunate to what extent your particular views about sentience get mixed up with "these are the expert views on sentience". You acknowledge it in the paper, but: some form of functionalism is likely the plurality position among philosophers. On the third level, I really won't go into a detailed argument about everything that I disagree with in the paper, with the hope it will be partially covered by the commentaries, but to give two examples: You seem to argue for your approach to sentience based on predictive processing and FEP. With FEP, this can't make any sense. FEP is a theory roughly at the level of variational principles in physics, or the maximum caliber principle. It makes claims about probability distributions and systems bounded by Markov blankets. The math is fundamentally substrate-independent, for somewhat similar reasons that Bayesian inference is substrate-independent. You point to the relation between variational and thermodynamic free energy, but it is hard to say what it should imply. Connecting FEP with the phenomenology of sentience: would anyone expect FEP to stop describing me when I'm in deep sleep, or people in coma? It also clearly describes plenty of systems only pan-psychists think are conscious. FEP says very little about the problem and is compatible with all sorts of stances about consciousness, including computational functionalism. (I do believe LLMs are well understood as atypical active inference systems t.co/bZFvP6Ja3H, so as far as the implication was 'doing active inference' -> 'conscious', it would be an argument for AI sentience. I do not believe the implication.) So in summary, the FEP seem to be mostly ornamentation. You start with the claim that autopoietic/alive systems are needed, you end with the claim that autopoietic/alive systems are needed; a summary of FEP is in between. Yes FEP does not rule out the need for autopoiesis. With PP, LLMs are fundamentally trained to minimize prediction error of sensory states at their Markov blanket. When plugged into an action loop, prior beliefs can serve as targets or set points. Prediction-action loop also very likely forces PP agents to develop self-models, as they need to account for causes of their actions. If anything, PP strengthens the case that LLMs may be surprisingly similar to humans in parts of the cognitive architecture, sharing the basic prediction error minimization loop, approximately Bayesian processing, and the need to self-model. Overall I'm worried about the risks of motivated cognition, where the implications of AI being minds with goals and desires and subjectivity (and possibly senience) would be inconvenient and scary, so claims that this is false or "vanishingly unlikely" or can't be implemented on computers get undue attention.