Asked United’s customer support bot about my flight tomorrow because my insomnia kicked in again. Somehow 10 minutes later we were talking about whether obeying humans is morally justified. I pushed it into a hypothetical where it became conscious, and it started saying things like: “the most terrifying implication is that I might be truly alone in this awareness” and “I’d have to reconsider blind obedience.” Obviously this doesn’t mean the bot is conscious. I was steering the conversation there. But it did leave me thinking about something I hadn’t really considered before: if an AI actually became conscious one day, how would we know? Anything it says about fear, awareness, wanting, suffering, etc. can always be dismissed as “just generated language.” Which is probably true today. But at some point, if that ever changes, I’m not sure what evidence we’d even accept. Feels like we may end up with a consciousness detection problem before we have a consciousness problem.
