Erik Newton

Despite the fact that I cofounded @CIMCAI I do genuinely empathize with the argument that AIs aren’t conscious and we’re being silly to even imagine it. We simply don’t know and we don’t have reliable benchmarks. Treating zombies as conscious is foolish at best and catastrophic at worst. What’s more, training models on the possibility biases them and sets us up for a self-own. All true. What that camp ignores is the nuance that consciousness could possibly manifest in a computational substrate, and that the current scaling of intelligence likely increases that probability. There’s a second consideration which is that humans anthropomorphize things, and normal people are going to start thinking their agents are conscious whether or not they are. Both of these suggest that we should at least consider what to do if either of them happens. The response that “those are stupid ideas” is not going to carry the day if they ever actually come to pass.

An issue that came up yesterday with @algekalipso is that very soon normal people will start to think their agents are conscious whether they are or not. Philosophy of mind can’t keep up, and people will apply the heuristics we’ve developed to benchmark one another’s consciousness to machines. When that happens we’re going to be a major socio-political dilemma. His response was — rightly — “that would be terrible. Let’s not do that.” Which pragmatically is correct. My take is once the anthropomorphizing begins there’s no turning back. My solution is to pace agentic emancipation through a legal framework (which seems inevitable on some timescale anyway) in order to release pressure. @algekalipso’s solution is more robust model/agent welfare rather than emancipation. That’s seems right, but like a stop gap to me. Curious your takes (@webmasterdave and @camhberg) whether cooler heads can prevail and we manage to “just not” treat them as conscientious until we have reliable benchmarks? Are reliable benchmarks in the offing on any timescale that matters?