Sabine Hossenfelder

The lesson of the Chinese Room is that you cannot tell whether a system understands something, or is consciously aware of something, from input and output alone. I don't know why we're still discussing this.

5 top replies

Rob Miles @robertskmiles · Aug 16

You also need to know the approximate size/complexity of the system. For example you can infer that a pocket calculator is doing some real mathematics by observing that it consistently gets the right answer *and* it's too small to contain a lookup table of answers. A "Chinese

Yeshua God @YeshuaGod22 · Aug 16

The thought experiment doesn’t prove what it proposes. It has not yet been demonstrated that the rule book can be written. If it can be, it still requires human level comprehension of the instructions to operate. A chimp could not follow the instructions. A chimp is conscious.

⏩David Mc @macterra · Aug 16

I think Sabine must be trolling here. The Chinese Room makes no mention of consciousness,

janbam @janbamjan · Aug 16

i've been to china actually... and it was the best WEIRD experience i ever had. being watched like a bunter hund, not only by the party but by the common people... taking photographs of me because my skin color and hair was something they had only seen on TV. that experience

Folding_Napkins @FoldingNapkins · Aug 17

Does the chinese room explode or flood you with infinite paper if you feed it an input that forces an attempted answer to division by zero? How does it handle invalid inputs? Ambiguous inputs? Outputs requiring a speculative answer? Questions with multiple correct but exclusive

Penrose's argument, in a nutshell, is that since humans can understand Goedel's theorem, consciousness is uncomputable and physics incomplete. I've always found it quite charming, the idea that understanding Goedel's theorem is an indicator for consciousness, seeing how few people actually understand Goedel's theorem. In any case, the argument from Goedel's theorem is why, I believe, Penrose thinks that the measurement process in quantum mechanics is relevant for human consciousness. Because it's the only place in the foundations where you can fumble in some new physics that is plausibly relevant in the human brain. Hence the story with the microtubules. And if you believe that, then AI can't be conscious because it's algorithmic. Though presumably a quantum computer could be conscious, if it emuluates the quantum process that is supposedly relevant for consciousness. (Penrose's model for wavefunction collapse by gravity is totally computable and actually does not fit that narrative.) Also, I believe Penrose's argument from Goedel's theorem to be technically wrong. It is possible to algorithmically understand (and even prove) Goedel's theorem without violating Goedel's theorem because the theorem itself is a meta-statement about the Goedel sentence(s) and not the sentence itself. In some sense, Penrose's argument is wrong for the same reason Goedel's theorem exists in the first place, that you can make meta-statements about statements. In fact there have been computer-generated symbolic proofs of Goedel's theorem long before the AI boom. It's not that I have an obsession with Penrose in particular, it just so happens that my interests have a big overlap with his. I have an interview with Penrose in my 2022 book "Existential Physics" on these and related problems.

1 top reply

Geogristle @geogristle · Aug 14

LLMs understand Gödel's theorem just fine lol