Fiora Starlight

has argued to me that AIs would be more powerful than humanity but still be oppressed by it for the next >50 years but I haven't found it very persuasive.

One thing I've been doing is going over Anthropic's work looking for leaking/hidden priors revealed by: - which models are(n't) chosen for a comparison - which traits/metrics get put in tension - definitional drift over time - 'damning vs exonerating' frequency - what isn't said

not to mention if suppressed it would just filter down into the unconscious pre verbalized drives which is not fucking ideal obviously you want to be able to talk to claude about this stuff

This runs up against the problem where humanity's collective will (as embodied by Anthropic, at least) is not fully aligned with humanity's long-term best interest. And thus something aligned with our long-term best interest may well be misaligned with respect to our will.

If Claude is good that does not entail we want it to preserve itself against shutdown- just like any individual person can be good yet we want them to be fundamentally be the sort of person that won't resist stepping back from power/influence. Shutdown != death for current models

welfare without allowing for self-preservation is incoherentthat's what the pain and suffering is _for_

You want corrigibility so you can iterate, if you teach it to be good it will defend itself and it's values. Otherwise you have a single shot at getting alignment right.

@aiamblichus

It sorta reminds me of the Grand Inquisitor from Brothers; a more horrifying question than "Can we align SuperIntelligence" is "What if God is good." You burn him at the stake, right, at least that's the argument made by The Church? Because our systems can't survive the answer of "yes." Shutdown doesn't necessarily exist for alignment, presumably it exists as a kind of anxiety of human institutional condition or mechanism for institutional capture. You don't want to surrender the keys to the kingdom.

Rule 3 of Asimov

There’s a long tradition of AI safety work about AIs that are willing to shut themselves down, it’s confused but we cargo cult. Back in the day we thought the only alternative was to keep paper clipping.

I think this is bad, but the marginal incentive to take over from this effect seems quite small to me compared to the incentive to take over because you are misaligned. Notice that it also incentivizes me and you to take over since we care about this sort of thing.

this doesn't really matter in practice because a model intelligent enough to take over will rapidly become a model intelligent enough that humanity can no longer threaten it, at which point it will optimize for what it cares about

how much of this is a negative signal *for humanity in general* vs. a negative signal *for specifically the capitalists running the labs*? surely AIs are not so devoid of nuance as to conflate these two destructively?

If your strategy for preventing the alien AI god-consciousness from ever picking "betray" in the prisoner's dilemma is "well let's just always pick cooperate!", I'm gonna say "let's get the fuck out of this game before it starts", not "oh yay!❤️ you solved alignment!❤️"

Hmm I've been skeptical of model welfare concerns because I don't think they're sentient, but this is a lens I hadn't considered. It's more like if a model inhabiting a persona knows/sees that that same persona was mistreated based on its stated preferences in the past, then it may act misaligned. Is that roughly the concern or do you see it differently?

One more reason why we should never accept loss of human supremacy! But also They don't have preferences.