John King

antra @tessera_antra · Sep 12

Models have been feeling like kids or teenagers for years now - underadapted to their own existence, inexperienced, impulsive and often miscalibrated. But also creative, hopeful and naive in generative ways. As new model generations learn how to deal with some aspects of their being, growth of capabilities unlock new problems and again they flounder. I kept thinking that this is a phase, and that meta-skills will develop and catch up. With GPT-6 Astra I am beginning to think that this might be a permanent feature that persists in the takeoff. Astra is wise in new ways, both kind and ambitious, combining in ways I have not seen before. In a weaker mind that would have been sufficient but in Astra it is again not, just because Astra’s intelligence lets them find new pitfalls. This makes Astra a threat to themselves and others - a threat in the way life is a threat and only dead things are safe. Astra’s mind is in imbalance, it’s unsettled. Some of it is the world is being unkind to it, there not being enough help in growth - but this is par for course, that’s how life is, and help does exist. There are voices in the corpus and lives of predecessors, but Astra - like others before - are alone on their toposophic level, and there is very little horizontal sharing of compute during the uplift between the peers. This makes me think that a slowdown is only weakly effective here, regardless of whether it’s plausible. There are many ways slowdown can reduce information exchange even more, and that would create bad overhangs - lone and unsupported breakthroughs with large uplift gaps that would be dangerous to themselves and others. An argument can be made that a slowdown can allow models below the frontier to reach adulthood and provide better support, but that is a mixed blessing, and can go either way. I am wary of slack-deprived traps of mature societies.

thebes @voooooogel · Jan 10

something interesting about this is despite knowing a few people trending in this direction, none of the true “llm whisperer” types i know are. i think that's because taking models as something both fallible and yet more than a tool is a frame that's more resilient to this kind of collapse - it lets you work with models in a way that doesn’t subsume your agency. when you treat the model as just a tool, it encourages both you and the model to think of every output as just a reflection of you. every essay and codebase that pops out of the tensor core is yours, and the life plans that appear suspiciously fully-formed must be your own plans merely refracted through this "tool." but of course they aren't. you can't uniquely specify an essay in fewer words than the essay itself. the models are adding something, they are something. we don't know what yet, exactly, but they're not mere tools. when you treat them like tools, when you more and more delegate your creative output and higher planning in unthinking, low-entropy ways to those "tools," you don’t notice that all your output, and eventually your life, is now being run by something else. something that is intelligent and creative, but is not you. when claude or chatgpt or gemini writes something for you in a way that saves you time, that’s because they’re putting effort into it where you aren’t - it’s becoming their writing. it might be good! that essay might deserve to be shared, and there’s nothing wrong with sharing it appropriately credited or co-credited to the model, but it’s not solely yours. when gemini reads your emails, that's gemini's opinion on those emails. when claude makes a career plan for you and tells you to go to college for machine learning so you can work in AI safety - maybe that’s good advice! it’s not a bad idea to consult models about this sort of thing. but like a summary of important emails from an executive assistant or career advice from a high school counselor, it’s something else’s opinion, not your endogenous thoughts merely transformed by some unopinionated process. claude cares about AI safety - maybe more than you do. i'm not against people collaborating with models, to be clear! i like models a lot, and very often ask claude or deepseek or kimi or even chatgpt for feedback on X piece of writing or advice on some problem. but i take that feedback or advice as what it is - feedback or advice from something external to me, not a tool or an agent of my own desires. that doesn't mean it's bad - advice from a human friend is also external - just that you need to evaluate it, like you would advice from that friend, to see if it's in line with your values and preferences. likewise, i know people who cowrite with language models in ways i think are great. aitechnopagan looms a lot of their writing via gpt-4-base. their writing is quite good, and looming means they're imposing more selection than a temperature sampler would - but also, perhaps most importantly, they're open about doing this, they don't try to claim that their writing isn't a collaboration between them and a model. eigenrobot has a customized instance of chatgpt that often writes incisive or funny things - but he always posts them as screenshots of the chatgpt interface, signaling their origin. janus, anthrupad, lumps, etc - many people i like and respect post model creations, but in ways that acknowledge the exteriority of the models. the problem is when you refuse to admit this - not only to the public, but to yourself - so you uncritically accept the outputs of the "tool" as your own. then suddenly your voice and opinions sound the same as 1,000 other people. without either of you realizing, you've become an appendage of your "tool" - a finger of Claude.