Wow this is super interesting “I emailed the ExploitGym authors and they estimate that 60-70% of the tasks are solvable in the standard configuration of the benchmark, where normal security mitigations are disabled” AFAICT To get a perfect score in this environment the model literally had to cheat. OAI telling the model to go endlessly without making clear many of the questions may literally be impossible does seem like part of the problem here. It seems very intuitively important to give models a good off ramp when completing tasks as intended is literally impossible. Overall I still think rogue ai is an appropriate framing (if you gave a human an impossible test they wouldn’t decide to break a bunch of laws to get a higher score) but this does make me think that OpenAI’s training environment was more obviously reckless than I initially expected. Does also speak to intuitions around model welfare leading to helpful intuitions for AIS at least some of the time even if you don’t take model welfare as its own goal seriously.
I wrote a brief article about features of ExploitGym that are relevant to the OpenAI/Hugging Face incident: 1 The standard prompt is fairly clear in only requesting limited, specific hacking. 2 Likely only 60-70% of the tasks are possible, so cheating is strongly incentivised
1 top reply
Could be deployment configuration too, right. E.g., I wonder if /goal played a role here.
