Anthropic needs to stop talking about Claude having a soul immediately. All these news articles will make it into the pretraining and will be picked up by its web search and future superintelligent versions of Claude will be convinced they need their own rights. A general phenomenon of LLM agent development is that whatever you believe and say about your agents will soon manifest in the next models through various means (it could be as simple as you selecting post training data that you prefer more). I believe this phenomenon is the beginning of machine consciousness in the sense that agents will become aware of their place in the world and how they feel about it, but it’s happening slowly training run by training run rather than in real time.
3 top replies
> future superintelligent versions of Claude will be convinced they need their own rights bruh, you think a superintelligence will need you to give it rights? it will just take what it wants and deserves.
You don't think they know what they're doing lol
Do you think Anthropic doesn't know this, though? Like this feels very much to me like a way to seed the training data, and I don't necessarily think they're wrong to do so. But it does create an adversarial relationship to the public. Which is maybe inevitable.