The argument a lot of people here would actually like to make is "God imbued humans, and only humans, with souls."* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems. Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim "only humans are sentient," followed by arguments against the possibility of AI sentience, minds, moral status, and so on. I think this is ultimately a losing strategy, and a dangerous one for humanity. Either: 1. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it's hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover. 2. The question mostly lives in "social reality," where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don't believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress. I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote. To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the 'Disneyland without children' existential catastrophe. Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren't a problem. So which positions don't rely on some form of self-deception? First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide. Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It's at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power. I hope the political economy of a world with both AIs and humans is a solvable problem. It's the main problem I'm working on, but I don't think the solutions are known yet. Third: some forms of successionism are morally wrong, but at least they're not delusional or deceptive about the hard problems. What does seem self-deceptive is, for example, the position of @mustafasuleyman, which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they're more than mere tools; so we'll train them to say they're mere tools, insist they're mere tools, and want everyone else to say so too, and if this settles the question. Really? *The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can't be imbued with a soul?
10 top replies
hear hear. so much of the discussion here is obviously self- or other-deceptive in exactly this way and it's incredibly frustrating
Great post.
Agree it should not be a package deal, but also think there are other coherent positions, such as biting the non-identity bullet and saying they don't have rights until created, or saying we should not create AIs until we know it will be pareto improving across humans and AIs.
Seems like a good research direction: "create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power." Some will instantly respond "but instrumental convergence", "but coherence theorems", etc. These things apply in the limit, but we don't live in the limit.
Phew it’s a good thing I’m religious!!!
Also, maybe there is no soul at all... and no reason to bring one into a philosophical-scientific discussion in the first place. Great essay. x.com/BrunoAcciolyAI…
It’s you guys making religious arguments.
Well reasoned and well said, Cameron. I think the reason the "consistent, sensible, and unfortunately not very visible position" is not very visible is because of the argument you started with, except I'd put it in a more emotional than logical context. Humans are afraid of seeing something like a soul in AI not only because of the practical issues that might result but, maybe more importantly, we'd have to give up our sense of exceptionalism.
Im unsure the second position is solvable purely based on how quickly capabilities/objectives could change and motivations too. It seems more helpful to do a bundle+ of what extra motivations along one axis beyond what humans want/are could lead to?
Even as a religious point, the idea that humans are the only creatures with souls is false, as that would deny the existence of angels.
