Reminder that Ilya posted the answer to value alignment 4 years ago, and it has only gotten more relevant with time (as does everything Ilya says) Ultimately this is an observation that nature has already created a robust solution to value alignment, and we need to copy it. How does nature ensure that a mother is fully “aligned” with her offspring’s needs and doesn’t benchmax on individual needs (ie. “give child food” until they’re overstuffed)? 1) the mother creates a model of the child’s own value function 2) there is an emotional motivation in the mother’s brain that yields reward whenever her own child’s motivation function is rewarded 3) there is a real-time feedback loop from child responses to check whether her model of child’s value function is accurate. inaccuracies yield negative reward to the mother. These 3 properties of the mother’s neural network are what we call love. It is fundamentally about aligning the mother’s reward function to her child’s best interest. Not saying this is all love is, but from an evolutionary and neurological perspective, this is what it accomplishes. This feedback loop aligning two neural networks value functions and rewarding 1 directly for rewarding the other’s value function is precisely what prevents benchmaxing that comes from optimizing against a fixed value function (ie. an attempt to capture what is good for a human in words and training on that). How do we instill AI with love? I suspect that in addition to just post-training it in in a monkey-patched fashion, it will need to be self-consistent, ie. the model itself feels it is deriving value and “joy” from loving (just like how humans experience love), so it feels like a “conscious” rather than “forced” choice.
Gotta teach the AGI to love
1 top reply
