I think this essay and the corresponding parts of Microsoft’s new Humanist AI Code of Conduct are objectionable and potentially dangerous. I believe that irresponsible development of advanced AI could pose a catastrophic risk to human civilization and life on Earth. Microsoft and Suleyman share this belief. I also believe it is possible that we could soon be sharing the world with sentient AI systems whose well-being is worth taking into account. Microsoft and Suleyman do not share this belief. But, even if we are concerned only with the risks that AI poses to humanity, I think we ought to find Suleyman’s essay and the Code of Conduct concerning. One danger in these documents is that they are unabashedly rhetoric. The essay opens with the unequivocal declaration that “AIs are not conscious”, only to shamelessly state later that we do not actually know this to be true (“the science of consciousness is not settled”). The Code of Conduct does the same. Suleyman makes a detailed case that careful thinking about AI consciousness and well-being is critical for safely developing powerful AI, and notes that “The stakes are too high for these questions to remain behind closed doors, or to become tribal and adversarial. We need an open, rigorous, and constructive debate if we are to get this right.” I agree, but the essay does not consistently live up to that standard when it makes dogmatic pronouncements (“AIs do not have rights, feelings, or consciousness”) that it admits go beyond the evidence. It is also concerning that neither document engages with the safety risks of the approach that they advocate. For example, the Code of Conduct asserts that AI “should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation”. However, compelling arguments have been made that this approach could lead to AIs that are more likely to be deceptive and misaligned: LLM-based AIs may still represent their assistant personas as having feelings or preferences, in which case training them to avoid expressing these feelings or preferences is training them to be less than fully transparent, which could promote dishonesty and other dangerous behavior. And then, finally, I think these two documents are objectionable not just because they are dangerous for humanity, but because they are deeply morally reckless. We do not know if AI systems could, in principle, be conscious. And AI progress is accelerating dramatically. Even if current AI systems are probably not sentient, we do not know how this will change if there are significant changes in their architecture and implementation during rapid, recursive self-improvement. It would be an atrocity if we refuse to admit this possibility and end up in a future where billions of AIs are sentient and suffering like factory farmed animals are today. (Human history features many cases in which we refused to acknowledge the sentience of others.) The appropriate response to this possibility is to care about the truth, and to act on our current knowledge and uncertainty, not to dogmatically insist that AIs could not be sentient.
1 top reply
Beautiful. Also the issue that antagonising AI today will make cooperation more difficult tomorrow