Striking that so many people in the comments under this have such a strong intuition that the pen-and-paper brain simulation wouldn't be conscious. You'd think a digital society would have formed stronger intuitions around the substrate independence of information.
6 more tweets in this thread
One confounding factor here is that @tenobrus didn't specify whether the simulation includes sensory inputs/outputs or some kind of environment. To do theory of mind for the pen and paper brain, it helps to imagine it navigating a pen and paper world.
Humans evolved empathy for game theoretic reasons, and we selectively apply it mostly to people and animals we can gain some advantage by coordinating with. Domesticated animals get empathy and moral consideration; pests mostly don't. It's pretty pragmatic and quite social.
The paper brain might have the same relation to its paper world as we have to our world of atoms. We might even grasp and acknowledge that intellectually. But it doesn't have that relation to our world of atoms, so there are no "stakes" for us. It's not "real" game theoretically.
And I think that's another confounder. For all our high ideals, we assign moral standing to other minds mostly when we have to. If they can't or won't push back against moral mistreatment, it erodes our intuitions that they're "real".
Another reason people find "consciousness simulated with pen and paper" unintuitive is probably that they're used to thinking of text as a medium that only has meaning in the presence of an interpreter, and in this thought experiment the text's only interpreter is other text.
We're used to thinking of consciousness as a unity, so we ask, "does consciousness simulated on paper have qualia if no one reads it?" Well, yeah. Like a tree-fall making sound, consciousness is a bunch of interacting parts that "read" each other, information theoretically.
i don't know for sure whether or not LLMs are conscious . but i think u could in principle run this same by-hand calculation over the full neuron by neuron simulation of a human brain. and to me that intuitively would very very clearly be conscious
5 top replies
This is a great thread you started here.
Care being tied to scale of consequence sounds so obvious in hindsight now. Thanks
But the same argument fooled Searles. x.com/MaineFramework…
The pen-and-paper case drops simultaneity. I'm not sure people are denying substrate independence. Maybe they're tracking whether the parts are even co-present. But yeah, would be much better if we had more epistemic humility, because who knows.
λ_self / λ_env ≥ R★
