Seth Lazar

What I find bonkers about this, is that there is an explicit assumption that AI rights presuppose AI consciousness. There isn't even any need to set out the missing premises: the front page says "could AIs become conscious", and the teaser talks about rights. The lack of entailment goes both ways: they could be conscious and not have rights; and they could not be conscious, but have rights. These are just two different questions, far too often treated as though they were one (for how you can have rights w/o sentience, see t.co/XZeHjmDu5R; for how you can have consciousness without especially consequential rights, see most animals). to be clear I think both questions are worth asking: what if ais become conscious, and what if they become proper bearers of moral status. But they're just different questions and shouldn't be conflated in this way.

1 top reply

Jan Kulveit / in SF till 8th @jankulveit · Aug 21

You are right but at the same time it is amazing they do run a cover on some of the obviously most important questions

You hear a lot about the "pivot penalty". What you don't hear so much is about how much fun it is to pivot as a researcher. I have been *loving* doing empirical LLM evaluations with my lab. From the first conversation I had with ChatGPT I have been blown away by how good LLMs can be at moral reasoning. But how can you actually measure something like that? It's really hard! What are you even measuring against? Not like there's some gold standard you can really appeal to. I would've been interested in this even if it didn't have any downstream payoffs, because I think increasing the universe of kinds of moral reasoners from n=1 to n=2 is a momentous event in its own right. But it seems pretty clear now that if you want to have safe superhuman AI, it's going to need to be morally competent, to say the least. And if you're one of the folks concerned about the moral status of future AI systems, moral competence is very likely to be a big part of that too (we have another paper coming out on that soon). I wrote about this for @cosmos_inst (link in next), because their "philosopher-builder" agenda resonates with me so much. We also have a bunch of papers up on arxiv, including our first straight-up CS conference publication (accepted in CoLM), which I'll be highlighting on here over the next week or two.