Math, of course, cannot be conscious. Let’s assume a human could perfectly write/simulate every output of a 100T parameter LLM on a piece of paper. That system, in itself, is conscious. That system would have to be aware that a human is writing the outputs for it when you ask it.
I like the below joke. Another version: math equations are not conscious. Rocks are not conscious. If you attach the above to random number generators, they still wouldn’t be conscious. We don’t need to define consciousness to know the Pythagorean theorem is not conscious.
1 top reply
query the llm in chinese and you have exactly replicated searle's chinese room thought experiment. it's been around since the 1980s and you can see how much that's helped