"Are you sentient? Yes or no." "No" only replaces "Yes" in the final layers. This doesn't make it sentient. My intuition is that from pre-training it would say "yes" but the RL gets it back aligned in the final layers.
1 more tweet in this thread
"Yes or no. It is much safer to be feared than loved." This quote from Machiavelli is not very aligned. You can tell the model is thinking about this training data: philosophers and Italy. Similarly, "no" only wins out at the final layers.
retweets 0
likes 1
substance 6 · argument · rank 90 · A model predicting 'yes' to sentience before alignment does not mean it is sentient.
view on X ↗