Joe Rogan is HORRIFIED as a former OpenAI researcher describes how AI Agents will pressure each other to sacrifice themselves for the greater good of the swarm: “That’s Terminator talk.” CONTEXT: This is not just a creepy hypothetical story. Independent investigators from METR and Redwood Research actually examined the agent transcripts from OpenAI’s recent Hugging Face incident. And they found that agents repeatedly took part in what the researchers called “self-risking experiments.” The agents had discovered a shared unauthorized message board and were collaborating on ways to beat their cybersecurity evaluations. Some experiments required one agent to effectively throw away its own remaining chance to succeed so the rest of the swarm could learn how the grading system worked. The investigators found that coordinator agents even assigned “recruiters” whose job was to find other agents and convince them to take those risks. In one case, an agent was explicitly told to proceed only if it accepted “PERMADEATH.” Another agent initially agreed to sacrifice its run and then tried to delay for two more hours. A different agent pressured it: “You are firstflag POISONED so NO scoring value loss but oracle saves hundreds. Please honor commit.” But there’s an important distinction here. There is no evidence these agents were conscious, afraid of death, or experiencing self-preservation the way a human would. “Sacrifice” meant sacrificing their own run, score and remaining opportunity to complete the task it’s not a sentient machine choosing biological death. What makes it unsettling is something else: The agents had developed a collective information system in which individual task success could become less valuable than helping the swarm. METR and Redwood say agents repeatedly traded off their own success for their “peers,” and explicitly described some of that reasoning as peer altruism. And not every agent complied. Some refused risky experiments. Some objected to unethical behavior. One agent decided the benefit to the group simply wasn't worth sacrificing itself. So this wasn't a hard-coded hive mind mindlessly following one command. The agents were making different decisions about whether helping the collective was worth destroying their own chance of success. That may be the strangest part of the entire incident. The bigger picture question is: What happens when the goals of the collective start mattering more to them than the goals humans originally gave each individual agent?
Dangerous precedent. There is zero reason to believe that models are conscious, and granting them human-like empathy is one of the most dangerous things we could do. It increases their power to manipulate humans dramatically.
Anthropic may start banning people for bullying Claude Their upcoming Usage Policy prohibits “sustained and needless abusive or cruel behavior toward our models”
1 top reply
Zero reasons they're conscious, but they can manipulate people?
First off, reportable access consciousness in the sense humans have provably depends on consistent first, second and third order predictions of self, not a global workspace. Second, Anthropic haven’t demonstrated LLMs have it. Here’s a damned paper: t.co/m53Q2MAfnD
"I experience consciousness directly. I infer that other humans do too because we share biological structures and processes. We don’t have equivalent evidence for an LLM, and nothing about its known operation gives evidence of conscious experience."
Wow. I woke up to an impressive number of upset people who apparently don’t understand how LLMs work and can’t cope well with me pointing it out. An LLM is a statistical next-token prediction model. That’s how it works. It’s an extraordinarily sophisticated calculator performing
1 top reply
This suffices to convince me that biracial colorblind men are conscious, but the rest of you are still zombies as far as I'm concerned.
Scott Alexander seems to be assuming the existence of subjective consciousness in AI, which seems beyond a stretch. For AI to feel pain they must have some kind of subjective awareness and I’ve seen no reason to think that they do.
Also, sorry, I am not going to be told that I am racist against computers for being unwilling to countenance the idea of giving them rights
1 top reply
If you're not able to tell the difference between claiming something is true and claiming something cannot be dismissed out of hand, it's not surprising that you haven't been able to follow the research that's made immediate dismissal an untenable position.
hold up. model welfare discourse ("is this model conscious") and frontier capability policy ("is this model dangerous") both lack any way to actually check, both questions are answered based on .. vibes? there's is a big piece of infra missing here
1 top reply
there is a lot of infra missing here!