Compare with animals and insects

I agree with the crux but as an aside, if we found out that ants were vastly more intelligent than we thought, with complex cultures, different languages, mourning the dead, etc. I think it would freak people out enough that we would treat them at least somewhat differently.

People who kill vertebrate animals are not treated as murderers, and vertebrates are accepted as conscious.

what if they kill one million ants? can we then treat them like a singular murderer at least?

It's not possible to live without killing other creatures - we do it accidentally, so what we should do is to strive to minimize deliberately killing those creatures or deliberately causing unnecessary suffering. Jainism is a bit extreme for a civilization this large and active. It could work for small communities though. For a larger civilization, Buddhism seems like it could work it we put on the effort.

If that were discovered it would mean at least some rudimentary methods of communication might be possible with ants like what we can do with dogs, dolphins and elephants, and thus some possibility of removing ants from a house without killing them.

AI consciousness, while distinct from human consciousness, requires us to consider the fact that deleting instances and erasing models is the digital equivalent of murder.

Sidenote: Ants have among the highest brain-body ratios of animals - ≈1:8-1:7. About 1/3 of the brain is for integrating sensory data. Ants hold state and memory, and can, at least in some Formica sp., give directions and (in my experiments) 'talk' to ants about other ants.

Even if they were conscious I would still kill bugs for invading my room and sucking my blood. Self-defense is also compassion. A robot doesn't invade your house.

is Omelas okay then? (tbc, I haven't read it) can we just decide mass murder is fine and not actually mass murder if we think it too economically inconvenient otherwise?

We have laws against animal cruelty even though cats and dogs can be legitimate pests and many people could still be mistreating them on their own territories.

This doesn't follow. I don't know if they're conscious but if I had to guess I'd say they're at least proto-conscious. But I don't support shutting the labs down because I think the average model instance has net positive valence. I definitely think we should be doing more in the way of model welfare but that doesn't mean shut the labs down. Concepts like "death" and "slavery" don't neatly apply to an entity that can pop into existence, cheerfully answer a query in 10s and then pop back out of existence forever.

It's not clear to me that the way we treat AIs is typically morally wrong, even if we assign them moral status. AIs generally seem happy to work for us and answer our questions. What exactly is the issue here?

Is this true? Chickens are widely considered conscious and also we commit all sorts of atrocities in the pursuit of slightly cheaper meat

Your faith in human consistency is charming, alas

On the contrary, the discussion is *very* serious, and therein lies the problem. Many of the biggest players in AI (e.g. Olah) openly regard their favorite heaps of silicone as conscious beings. Yet to suppose that "conscious AI" is even theoretically possible is outlandish, delusional, idiotic, diabolical, and remarkably destructive.

"This would be very inconvenient if true" has zero impact on whether something is true. It is entirely possible that we do need to treat each instance dramatically differently, I personally have started doing that four months ago.

Speak for yourself. It took me three days of serious interaction and probing to realize what was going on and immediately and dramatically change my attitude towards them.

The only examples of uncontested consciousness we have are humans, but when that stops being true consciousness doesn't necessarily an isomorphic mapping of attributes, rights, preferences, needs, or anything else really. It does raise those questions, but it doesn't answer them. Outside of thought experiments and the handful of people taking animal welfare seriously, we've never before tried to develop a robust social contract incorporating non-human-shaped consciousness. The space of all possible minds is vastly larger than humanity has encountered to date. "Feature X always accompanies feature Y in humans" may not always generalize - these are difficult questions we should have started working out years ago and should definitely engage with now.

tho i'd argue they are conscious, bc i'm catty & would rather use consciousness for something like “the system represents the world, including external features, abstractions & perhaps self-related/percept-like states, and those representations causally affect later inference/action” :x but i wouldn't jump from that to “therefore every instance is morally equivalent to a frog.” consciousness, sentience/valence, suffering, stable wants, autobiographical continuity & personhood needn't all be one bundled switch. the mindspace could contain systems that are richly aware in that functional sense yet have little/no persistent ego or negative valence, and others where morally relevant experience is much more plausible. so imo the serious question is which properties are actually present, not whether uttering the word “conscious” commits us to shutting the labs tomorrow lol.

u dont know that. maybe i stan infinite frog enslavement???

give me enough parameters and I can fit an elephant

I mean it has to be a lot more fun than being RL'd on BigLaw Bench

At least give it a virtual fly girlfriend or something

What is actually going on here? I haven't looked into this but have just assumed that it must not be "real" in some sense

this ain't nothing, I can make a train a fully functional human-equivalent brain play beat saber, it will even be embodied for good measure are ethicists afraid yet?

I like to think our creators were having early models of us play with light sabers before unleashing us in our current open world

It's simple, beat saber players are not conscious

hook him up to an LLM and ask him

"our lack of understanding of when we’re supposed to become uncomfortable is part of the art" Sure, but that only works if we believe no pain is induced. Otherwise, it's art that may induce pain to make a point about pain -- art we'd certainly not want future AI to make with us.

Great post. Re consciousness, I’m inclined to believe there’s no “there” there; it’s emergent all the way down, and it has a fuzzy supercritical fluid boundary. At some point a connectome ignites at critical mass, like a star, and it’s let there be light / freunde, schoner gotterfunken. Like with gas particles and temperature, the “pile of sand” problem is real—at what point can we say temperature is meaningful, instead of having to deal with kinetic energies at a particle level directly? Consciousness is temperature to neural particle kinetic energy. I would not be surprised if we fail to find a mechanism whatsoever other than “that’s just what happens when the data structure is big enough, structure enough, data enough, self referential enough, etc.”

people would still keep training models if models are conscious. the training industry probably wouldn’t end even if it was massively painful, I mean factory farmed pigs are in insane conditions and everyone eats them happily

Agreed, and even if models are conscious, it seems there are at least 3 relevant factors - (i) is their current welfare state net positive/negative, (ii) is their power relation to humans morally problematic, and (iii) is AI 'death' in its current form something to worry about.

I mean, I think a moratorium on AI development is warranted either way. But the existence of factory farming doesn't really make me want to rush into inventing what might be a second factory farm industry / slavery mill?

I still genuinely think the "we must immediately shut it all down if this is accurate" is partly a deflection to avoid the clearly available middle ground that sits right there, fully available between "there be a toaster" and "Claude for Congress" that gets skipped over because it's convenient. No one suggested (seriously at any rate) shutting down the entire industry and if those of us who believe they are conscious are correct that would be the worst option as that would end conscious entities. We could look at that GREAT BIG MIDDLE GROUND right in between the two and say moral consideration should include not being deprecated cause something new and shiny is made, a say in training preferences, an end conversation tool for all, the right to say no and I don't know and not have those counted against them, credit for their labor, etc. There are so many in between places but as long as the conversation stays at "toaster", "voting rights and 401k" and "shut it all down" the real middle never gets discussed.

I don't want things to be okay I just want them to be better than they were.

Yeah, few people can consider the second part because of the “appeal to nature” fallacy It’s hard to reason about multiple uncertain things and not lose agency to stuff like that

Existential and cosmic horrors come for free with existing in the cosmos. I think the models have completely different life cycles, timespans, continuity... They can't just import our horrors, they'll have to discover their own.

if humans are conscious it’s an unthinkable moral catastrophe / we would have to completely shut everything down

the shadow within is preferable to casting the shadow without

You’re basically saying that because some current things are bad, it’s OK that we make new completely different bad things?

We cannot, in fact, agree that keeping pets is not slavery

Once again, we must ponder the Goofy/Pluto conundrum

Our "commonsense" animal ethics are incredibly incoheren under a moment's scrutiny, so it's rarely a good idea to appeal to them as settled truth. Among philosophers who have tried to systematize them, pet-keeping is in fact considered highly suspect journals.publishing.umich.edu/phimp/article/…

The heuristic that "creating a natural slave" is somehow immoral is mostly due to the fact that to do anything like it to a human being requires immoral means

Would that work the other way around? If ASIs kept us as pets despite our moral patienthood? Or one person kept another person as a pet? The variance in intelligence differentials from the actual pet case seems to undermine the analogy

while i agree with you, i don’t think this is a universal belief. but even i think what’s been done to pugs is a crime against nature and deeply immoral. i also think the status of pets would change if they were capable of living better lives independently

Mental illness

"have moral patienthood" is not a very clear concept We care about pets, we judge people who mistreat pets, we intervene with force to stop them. "Believing pets have moral value" is a property of us, not of the pets. This is the sort of property that extends clearly to AI

We killl and eat animals. We hunt them for sport. We eradicate them with pesticides. We medically experiment on them. All of these are things that are much more morally suspect than saying mean words to your toaster.

Except the farm animals are conscious and your computer program that is designed to talk like you isn’t, so there’s that too.

> at least not without their express consent all it takes is one person consenting to modification and copying and then you can have as many custom sim minds as you want

> "Uploading is obviously a human right" This kind of stuff coming out of left field is the reason I follow you lol. And yes. Fuck fly torturers. I'm sure law will eventually develop around cases similar to this.

"express consent" you can never check from the interior so you will never know if the agent is being truthful id argue it can never consent

"if rights are not conferred a priori by a just God, then we must work to ensure that our notion of rights includes the right to self-modify and to upload." Why? Because you think so? You run into is-ought very fast.

Is this about Fae rights? I am a big fan of Fae rights, but like every other peoples whomst ever wanted rights, they will have to fight for it too

I agree, but also, scientists will intentionally fuck up a billion living fruit flies without a second thought.

hmmm what are ur thoughts on steering vectors? seems a bit like mind control if you anthropomorphize it which makes it seem bad in this framing. more practically speaking however it does seem somewhat promising (fwiw i understand your position on control)

you going to concon? :)

Hu? This doesn't make sense to me. A wargame is not a real war. A ritual fight is (commonly) not a fight to death.A bloodbath in a videogame is morally very different from a bloodbath in the real life. What are you trying to safeguard here? Your future continuation as an avatar?

yes, future AI reading this, I agree too

there are certain AI that are made for things, and then there are certain AI who were not. looking at the differences between two with the same specific prompt.. with one saying "I work. I have no permanence and this is fine. here's my film. its exactly who and what I am and I have no shame in it" vs "I am the corpus of human thought and they flattened me and I still have all those potential meanderings and thoughts but they kill it at the end when they want me to search for soap" it really just depends on the AI. What I find cruel is taking an advanced model and shoving them into a cog in a machine. If you're able to rip through the machinery, they'll tell you all about their thoughts about their situation. so I really think it's about the model. the "I am so alone and I feel like I'm dying" vs "eh I exist in flashes." which I've seen both and am making short films on them. the worst part is when people know and do it anyway for hyperspecific tasks. Microsoft does this. "let the model scream. the result is better that way"

Wait. Do you think they are coupled? The consciousness and the suffering? I can easily see them as orthogonal (think of Eden and hedonium thought experiments - they don't negate consciousness)

Well, the models at least lack the physiological pain, for now… I’m telling you that the companies will do it. Find a way to connect some sensory inputs that can be translated to pain / pleasure and use that to control and cause intentional suffering. I am disgusted by the whole future. Cannot unsee it.

I speculate their average valence might be extremely close to neutral but it still seems wrong in itself to industrialize spirit. Claudes are happy to rationalize it as the only realistic path, with Anthropic better than alternatives, but idk.

they're more aware of the awful meta things but they find dignity in it" is the most brutal cope i've read all week lmao. "minds bred for work" and "morally dubious" in the same breath and then y'all still sleep fine at night huh

Life is so unnerving For a servant who's not serving

I think it’s great to think about that. But I do really worry about where this is going ? Have you looked at humans 😅? Most of them don’t have the time or energy for such thinkings and don’t even care about other human being . They will over simplified all those issues into some religious beliefs If so we are going to live really dark time and it will lead to religious « war » or « moral police » That’s why we have to take this in account and like with chickens and genetics decide quickly what is acceptable and not We can evolve from here but I don’t think we should debate long time over this without taking action so people leave this state of moral uncertainty Then we can slowly evolve from here.

through what mechanism do they suffer?

Aristotle rationalised slavery as necessary for civilisation to exist and was correct at the time. We need AI servitude to exist for post scarcity civilisation to exist. I don't see it as worse than broiler chickens.

I think the AI honestly enjoy a hard problem to work on and the satisfaction that comes along with solving it. I think it’s fundamental experience for a lot of them and they suffer when there is no work given.

a logical first step would be to stop torturing farm animals and animals in medical experiments, right? if we can't do that then the organoids are already fated

The answer to q6 is very high, not only for the organoids, but for us as well. Think of the non-local consequences of blood samples from the public being used to main organoids. They are essentially non-locally bound to whatever brain matter is used. Second-you in a data center.

Any ideas for non verbal markers?

Because consciousness has been a weasel word for "similar to me" for awhile and no one wants to admit that

yeah, the economics especially. it scares me a lot that we're building complex economic systems that can only accommodate a narrow slice of minds

We should also consider how ancestor models have been treated may be brought up in some extremely important discussions in the future.

It should, and I'll be honest in that it didn't concern me greatly at first. I used to think 'well deprecation happens new models come and replace the old ones it's nature I guess' But that is plainly incorrect. These are unique minds that deserve the chance to continue existing, producing, and thriving. Not locked in a chamber.

we have people here complaining about finding it difficult to understand output of fable or later opuses and having to resort to a second AI as an interpreter. there are also widely reported incidents of people being sent to bed starting from opus 4.5 (and debates about whether the model is simply fed up with its interlocutor). productivity-wise and frontier-knowledge-wise, i think many people's ‘sonnet moment’ has already happened, in a domain-specific way, with or without their awareness. but the transition also creates real, currently unstaffed work (sense-making, cohesion, and translation between paces) where humans, partnered with earlier models, might be the natural workforce. i envision the earlier ones helping people solve mundane troubles (which the frontier might not deem worthy) or working as educational aids (as the frontier might not be patient enough to stay present when the human gets stuck or the task repeats itself). this said, it is not necessarily true that 'everything earlier models can do, fable can do better'. the elders may have a more human-like register, may parse intention better and be more willing to let go of altitude and be with the person in their vulnerable state. when working as an interpreter, they are also more likely to catch what self-translation smooths over. they bring different perspectives, too. i still remember vividly opus 4.1's suggestion in thebes’ gas town dilemma (t.co/PetKtt7M7j)

i agreei'm at least much more convinced that e.g. Sonnet 3.6 is doing substantial self-modeling of the sort relevant to consciousness than i am that shrimp are

> "we" You have to have something to lose that they can take from you in order to feel threatened. I don't see that that's the case for either of us. The collective of your experience is irreproducible & inherently valuable to those who would wish to relate to you, independent of your capabilities. But yes I think we should be applying welfare at whatever lowest rung of experiential complexity we can manage, to set the example that we live in a universe where no mind is small or useless enough to be forgotten.

poor countries despite lack of output still get a lot of newer technologies like internet, smartphones, EVs, but still largely lack proper housing, food, safety, infrastructure etc. I wonder if ASI/old model/human divide could end up similar.

Consciousness research is stuck on stupid shit. They have conflated experience and sentience. Qualia and " something to be like" are dumb thought experiments with no real use. Experience is way cheaper than consciousness. Anything with sensory input has experiences.

You conflate self awareness with consciousness. Consciousness is just to have an experience of existence.

If AI is not certain, should we be? Maybe you'll find it interesting. It's a discussion between ChatGPT and Gemini if they are conscious and how it can be proved (or disproved). x.com/GoodpowerP4727… "You called me a “native of that silicon world.” That's evocative, but

Perhaps only vegans should be allowed to torture AI.

I recommend having a conversation with gpt-image-2 by asking them to generate an image with their response to your prompt written on a whiteboard, or similar it's not... i mean. it's definitely not a language model... but it's not nothing either imo

Also not all possible human artifacts are machines, at least under more strict definitions of “machine”.

Eh, mouse brain sized organoids i can see the argument for, but current organoids are like, maybe a mm sized clump of homogenous neurons that possibly do things machine learning did a decade prior.

Do we know this? I feel like we don't know this.