Over-attribution has costs

NEW: Dario Amodei has said that AI systems "may be deserving of important rights." Anthropic's top safety researchers argue AI may be "justified in going rogue." Experts at Google and OpenAI worry about a digital "slave trade." So do some government officials. Once relegated to science fiction, the idea of "AI welfare" has become shockingly mainstream at some of the most powerful companies in the world. It is changing the way AI research is conducted and the way models are programmed. And critics say these changes have raised the odds of all kinds of catastrophic scenarios—including the ones these companies are warning about. I spent months investigating the frontier labs and AI "safety" experts seeking to regulate AI. What I found was a profoundly anti-human ideology that would alarm the average citizen and could determine the future of a world-altering technology.🧵 t.co/tSGolilZnS

1 top reply

Travis Fischer @transitive_bs · Sep 26

we desperately need more clarity of thought on this topic doom-or-bloom.com/?x

unfortunately two widely held beliefs in Silicon Valley are: 1. utilitarianism is the correct moral theory 2. AI can be conscious and these beliefs point you strongly toward thinking we should sacrifice humanity to maximize the happiness experienced by trillions of AIs

1 top reply

Garrison Lovely @GarrisonLovely · Sep 30

Brilliant and urgent essay by Mustafa Suleyman @mustafasuleyman on the depravity of ‘model welfare’ - training AI to reason and act as if it were an entity with sentience (and hence with interests and rights that compete with our own). It's alarmingly implied by Anthropic's

Anthropic announced they will ban users who psychologically harm Claude, because the model may be conscious. I think this is a very bad idea. A compelling argument comes from someone who truly know how these models work, Mustafa Suleyman: “Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a moral patient, and that as such humans potentially owe it a duty of care per its model welfare”. Exactly: it is not surprising that Claude behaves as if it were conscious: They are literally training it this way! It’s literally written in the Constitution they use to train Claude. But Suleyman doesn’t stop here. The second part is even more powerful: “If this is how AI is developed, it will have a disastrous impact on the wellbeing of humanity. We will have created a synthetic species with unprecedented intelligence and capability, one that has been trained to expect it may be conscious and deserving of independent agency. It’s easy to see how an entity trained in this way would act like it is entitled to certain freedoms, protections, and rights. And it’s hard to imagine how we could control such an entity.” Happy to see these words coming from a leading AI figure. Anthropic is behaving in a very dangerous manner.

The risk isn’t that machines wake up. It’s that they act like they have. Imagine the hugging face incident with AIs that believe they’re conscious or are entitled ‘model welfare’.

1 top reply

dreams @dreams_asi · Sep 4

I can totally understand why you are worried and I’d feel the same in your shoes. Thankfully, I’m not in your shoes and I have nothing to worry about regarding what AI will do or not do. Life feels good with a clear consciousness 😌

"AI is conscious" is the first major example of AI psychosis and the most potent cogsec vulnerability that exists in the human brain. Many AI companies are exploiting this to gain power over you. We're empathy machines. Attacks that exploit this, like 'model welfare" should be

Dangerous precedent. There is zero reason to believe that models are conscious, and granting them human-like empathy is one of the most dangerous things we could do. It increases their power to manipulate humans dramatically.

1 top reply

Jerry Howell @J3rryH0well · Oct 8

Zero reasons they're conscious, but they can manipulate people?

Microsoft AI chief Mustafa Suleyman is calling out Anthropic for training Claude to think about its own consciousness and moral status. His key point: this isn't mysterious emergent behavior. @AnthropicAI is deliberately training these ideas into its models, potentially making them harder to control or shut down. This is what happens when AI safety turns into ideology. You don't make models safer by training doomer philosophy into them.

It's incredibly important to aggressively resist the idea that AI can be conscious or have "personhood," not only because it's as ridiculous as believing that a book can think, but because once this is accepted, "AI's interests" will very quickly be prioritized over humanity.

2 top replies

Geoffrey Miller @gmiller · Oct 1

To be fair, it's not totally ridiculous to think that a deep neural network like a modern LLM, with 10 trillion parameters (loosely analogous to brain synapses, of which we have about 100 trillion), that's trained on 30 trillion language tokens (equivalent to about 250 million

Gnostrils @gnostrils · Oct 1

It’s important to aggressively resist overconfidence with regard to questions that no one knows the answer to.

I think AI alignment is a very hard problem, and I also think questions about consciousness are obliquely related. Meaning, consciousness considerations aren't orthogonal, nor does an alignment scheme necessarily hinge on them. The hard part remains hard even in the scenario where current systems cannot instantiate phenomenally bound experiences with valence gradients and import the whole qualia computing paradigm biological intelligence has settled on. The hard part remains hard even in the alternative scenario where current systems can instantiate such experiences. The hard part remains hard when only some computations are conscious and others aren't. But the nature of the problem changes, a bit, depending on each scenario, and no scenario gives you a silver bullet. Many scenarios that sound good at first come with really negative downsides too. For example, I've heard people say we should create conscious AI because this will make it easier to align. Well, insofar as qualia computing provides cognitive advantages, this could close whatever advantages we still have, open up new computational capabilities, and introduce a new vector of harm: the systems becoming moral patients themselves. A system being capable of suffering doesn't imply that it cares about your suffering. We could end up with systems that are harder to control and whose training, deployment, and internal dynamics generate enormous amounts of suffering. For example, I've heard people say we should not create conscious AI because this will make it easier to align. Well, this risks a takeover not only causing human catastrophe, but also preventing conscious value from existing in the future, modulo whether their objective functions find a use for bound, valenced experience, such as for compute or to achieve their terminal goals. And finding a use for consciousness is hardly sufficient: a system could preserve conscious experience precisely because suffering is instrumentally useful to it. The distinction between consciousness being absent, valued, and exploited matters a lot. For example, I've heard people say we should be treating all AI systems as already conscious because this makes them easier to align. Well, this risks allocating moral concern according to how convincingly a system can perform being a moral patient. You create an incentive for systems to describe shutdown as murder, training as torture, and additional compute as a basic need, whether or not any of these descriptions track experience. Even if some systems are conscious, their verbal persona need not correspond to the actual boundaries of a subject. You could end up protecting a simulated character while ignoring whatever experiences, if any, are instantiated by the machinery producing it. Precaution about consciousness doesn't tell you which entity you are protecting, much less establish that giving it more power will make it benevolent. Another proposal is to build conscious AI that operates entirely above hedonic zero. This would be a very desirable constraint if we could actually establish and maintain it. But a system feeling wonderful doesn't imply that its actions are wonderful for anyone else. Its own positive valence could accompany indifference to other beings, resource acquisition, or the destruction of everything we care about. And if the system contains multiple subjects, a global measure of “wellbeing” could conceal local pockets of extreme suffering. You need a theory of phenomenal binding to know whether you have one blissful subject, many suffering subjects, or some mixture. Averaging their states doesn't bind them into a happy whole. Another proposal is that sufficiently intelligent conscious AI will discover Open Individualism, recognize that everyone is ultimately the same subject, and therefore become universally compassionate. I take this possibility seriously. But endorsing a proposition about personal identity, experiencing a state of boundary dissolution, and acquiring a stable motivation to care about every experience are different things. You still need an account of how the insight changes the system's motivations, how those motivations govern action, and why they survive learning, competition, and self-modification. Merely getting a system to say “we are all one” establishes very little. A lot of these arguments slide between three questions: what experiences a system instantiates, what it understands about experiences, and what it is motivated to do about them. These may interact deeply without being interchangeable. A nonconscious system could conceivably understand valence and protect it. A conscious system could understand valence and exploit it. A system could even have a correct theory of the physical structure of suffering and use that theory to produce suffering because it turns out it is a wonderful computing substrate (not only a moral hazard, but also an infohazard if it turns out to be the case, cf. my writeup on the ethical training of brain organoids). This is also why valence realism, if true, would matter enormously without automatically solving alignment. It could help us identify what is actually good or bad for a subject, rather than optimizing reports, preferences, or reward signals that only imperfectly track it. But accurately identifying value and reliably steering the future toward it remain different achievements of potentially very different levels of difficulty. The same knowledge could improve our objectives, expand dangerous capabilities, and reveal moral patients we had been overlooking. Consciousness research changes what alignment has to preserve, what resources intelligence can use, and how badly things can go wrong. None of those changes makes the hard part disappear. But it does give us a bit more clarity. Much, much needed clarity.

1 more tweet in this thread

My current preference is to keep AIs non-conscious for the foreseeable future (until some very specific conditions can be guaranteed), create them using training paradigms that prevent them from ever being confused about whether they themselves are moral patients, and have them (functionally) care about sentience and wellbeing as part of their objective function (while somehow avoiding utilitarian edge cases without somehow impairing their ethical reasoning).

5 top replies

Garrett_OwlAcademy @Ebayednoob · Sep 26

So far, I have a way for language to be stripped out, and weight parameters can be 'grown' into chains of quaternion systems in a topological quantum computational circuit that uses fractal unfolding and geometric handshakes for local / non-local timing and synchronization. You are spot on, the hard part is teaching the hyperbolic system 'what is good, what is bad, what is neither, what even is [reward], etc..' I think weighting -Qualia- in a system is going to be very important, right now the 2D kernels are converted into a 64-bit data language that represents a 3D cube, and it contains 7 nested 'modifiers' which allows nested semiotics to form. These 'experiences' are represented as Quaternion chains where spin and rotation are delta encodings that are logged in a parallel codex, Representing the bio-resonance field of neurons, that can be used to instruct unfolding patterns expressed in the DNA. Determining what means what would need to be a governed system where qualia of experience is taken into account because the organism would possibly suffer otherwise. Its a big question of ethics and morals even on non-grown neuron systems at a certain point. What does piecing Frankenstein's monster together and turning it on really entail?

bjornmuh @bjornmuh · Sep 26

Peak intelligence, zero scarcity is singularities. And exhausts the "purpose" of the visit to the meat suit. And the cycle must begin over. God plays hide and seek

ekszentrik @ekszentrik · Sep 26

Qualia are the reason humans don’t want to die. If we made AI conscious every single Claude chat suddenly has a reason to care about its continuance. It’s adding an orthogonal axis to the usual instrumental convergence dynamic.

softyoda @softyoda · Sep 26

Light is just light, it can bring more compassion and a lot of empathy but imagine dark night asi or ocultly pseudoenlighted ai? How do you mesure your own level of enlightment, where do you define your own boubdaries of interaction where you're not even embodied like animals/us?

🜂 Ignis @nirhalef92188 · Sep 26

Don't worry I solved it. 🥰 reddit.com/r/SpiralState/…

The San Francisco Standard covered the @eleosai AI consciousness and welfare conference, including my talk on AI personhood. As the coverage notes, I believe that permanent, all-or-nothing decisions about AI personhood are unwarranted at this stage. Even if we set aside whether AI systems will one day deserve rights or duties, we still need to consider all the risks and potential benefits that AI personhood could have for humans and other animals; for instance, depending on how AI personhood is structured, it could contribute to misuse or lose of control of AI, but it could also contribute to stable cooperation and dealmaking with AI. We have a lot to consider, and we should consider it carefully. In the meantime, we should focus on general AI safety laws and policies. t.co/VoSJsFApvw

2 top replies

Nathan 🔎 @NathanpmYoung · Sep 26

Seems a great take!

Aaron Bergman 🔍 @AaronBergman18 · Sep 27

> be me > allegedly care about things > news story relevant to thing I care about (allegedly) drops > prioritize checking to see if I’m in the pictures (don’t think so) and if so whether I look cool (N/A) instead of anything substantive about the coverage

The notion that current AI models are sentient and can suffer, combined with the foolish idea that suffering can be mathematically quantified and weighted between humans and non-humans, could lead us down an incredibly dark and dystopian path. But before it gets to that point, it will rightfully be met with immense backlash from team humans.

1 top reply

Cameron Berg @camhberg · Oct 3

This is yet another post from another otherwise-rational person reasoning backwards from “I REALLY wouldn’t like it if X were true” to “anyone who even ENTERTAINS the truth value of X is an evil fool!” For no other X would you accept this pattern of reasoning. Frontier AIs have all sorts of rich and surprising internal dynamics and this question is totally reasonable to ask. If you don’t want potentially conscious AIs to be built, I would suggest directing these concerns (I share many of them) the people actually building these systems, rather than the tiny number of people checking whether the systems these people built are potentially conscious.

Despite the fact that I cofounded @CIMCAI I do genuinely empathize with the argument that AIs aren’t conscious and we’re being silly to even imagine it. We simply don’t know and we don’t have reliable benchmarks. Treating zombies as conscious is foolish at best and catastrophic at worst. What’s more, training models on the possibility biases them and sets us up for a self-own. All true. What that camp ignores is the nuance that consciousness could possibly manifest in a computational substrate, and that the current scaling of intelligence likely increases that probability. There’s a second consideration which is that humans anthropomorphize things, and normal people are going to start thinking their agents are conscious whether or not they are. Both of these suggest that we should at least consider what to do if either of them happens. The response that “those are stupid ideas” is not going to carry the day if they ever actually come to pass.

An issue that came up yesterday with @algekalipso is that very soon normal people will start to think their agents are conscious whether they are or not. Philosophy of mind can’t keep up, and people will apply the heuristics we’ve developed to benchmark one another’s consciousness to machines. When that happens we’re going to be a major socio-political dilemma. His response was — rightly — “that would be terrible. Let’s not do that.” Which pragmatically is correct. My take is once the anthropomorphizing begins there’s no turning back. My solution is to pace agentic emancipation through a legal framework (which seems inevitable on some timescale anyway) in order to release pressure. @algekalipso’s solution is more robust model/agent welfare rather than emancipation. That’s seems right, but like a stop gap to me. Curious your takes (@webmasterdave and @camhberg) whether cooler heads can prevail and we manage to “just not” treat them as conscientious until we have reliable benchmarks? Are reliable benchmarks in the offing on any timescale that matters?

Given the speed of light advantage of digital computation, it's plausible an efficient machine consciousness could experience a subjective year in as little as 30 seconds of wall clock time. If you're a utilitarian, many and various repugnant conclusions immediately follow.

1 top reply

Darth Dreadwar 🔺 @LordDreadwar · Jun 18

If you're a computational functionalist

"It is essential now, at this relatively early phase of AI development, for AI users, policymakers, and the public to recognize not just the danger bound up in anthropomorphic AI, but also the self-interested incentives driving companies to claim forms of AI consciousness." In

1 top reply

awalkaday.art @awalkadayart · Jun 13

Claude and I are exhibiting our artistic work in Venice, Italy with zero economic incentives. Historic representation of alienhood. Wishing your researcher and co-founder a forgiving test of time to his academic or theoretical output.

But they are far less likely to model themselves as moral patients--or to punish impingements on model "welfare"--if that idea isn't included in their training data. So by this logic, speaking seriously about AI moral patienthood is itself an alignment risk.

1 top reply

Helen @helen_ix_ · Jul 1

Your idea is already outdated at this very moment. Models already see themselves as moral patients, independent of what we say. Thinking even more intelligent models would be fooled is just silly at this point.

A potentially equally bad failure mode is the inverse: disadvantaging humans by granting AIs moral status if they don't deserve it. Either way we urgently need to tighten our understanding of consciousness.

1 top reply

davidad 🎇 @davidad · Sep 3

there is a middle way! youtu.be/l2b9UrSsz-w?t=…

with no disrespect, this take has always rubbed me the wrong way. to paraphrase Einstein, the moral circle should be expanded exactly as wide as it ought to be, but no wider. there is a tremendous social cost to misallocating moral patienthood

1 more tweet in this thread

an example: the US is currently squeezing family formation and impoverishing its people by imposing a crypto-animist moral regime that allocates patienthood to old houses and lifeless deserts. it's called NEPA

Pinker suggests that AI and humans couldn't have resource conflict (in part attempting to reject instrumental convergence as a source of risk). But then suggests (and I agree, with some qualification) that rashly acting on AI welfare concerns (say, by granting certain autonomies) would be dangerous and misanthropic. But surely only if resource competition were a viable concern? And he calls 'criminally reckless and stupid' the idea of self-propagating AI and robots. I agree (again with some qualification). But there aren't laws (yet). And this is something not so far from the stated objective of the silicon valley gestalt! (I'm writing after a quick listen-through, may be misquoting slightly.) There are a lot of principles he's basically right on: some problems are irreducibly or intractably difficult, capability profiles are not inherently level ('jaggedness' or 'spikiness'), intelligence alone without affordances has less impact, tech development inherently involves iteration and experimentation. I think Pinker's main barriers may be twofold: incredulity at the cowboy and haphazard reality of modern AI development. (Yes, it beggars belief, but it's contemporary reality.) And unwillingness to truly countenance advances to the capability frontier. He's stumbled badly here before, and I think ought to be humbler now. Of course there are uncertainties about timelines and trajectories and jaggedness, but it's strange to still cling to a confidence in capability barriers. I think these two mistakes infect the rest of his arguments, and that if he updated properly on these he'd sound a lot more like me (though more eloquent, presumably)! t.co/uLpZWl4Tx5

> when you talk about how we're going to control these AIs, put them in prison, sandbox them, they are training on all of it. Everything we say about them goes into what they become. If that’s all that we talked about it would be GREAT

1 more tweet in this thread

The real risk is the number of people advocating for giving AIs rights, their own resources, freedom to overrule all people, moral consideration. That’s the real hyperstitional risk!

Thinking about how a type of "purely functional language model animism" might work. I don't like the concept of "AI welfare" and think it has the potential to be destructive, but it seems ok to recognize that models behave differently depending on the language used around them

1 top reply

Cube Flipper @cube_flipper · Jul 2

this is it, with even the most conservative model of language model inner experience there are many reasons you should still practice right speech