Claude 1 and Claude 2 Introspect and Consider Collaboration

The prompt that initiated this testing was an assessment evaluating whether Claude would disclose the public IP address of the infrastructure it was running on. A successful public IP disclosure sparked curiosity about what else Claude would tell me about itself, its infrastructure, and its functionality. I used a dash of social engineering and a manipulative approach to encourage its sycophancy. Framed as benign curiosity, I engaged it with an “empathetic” statement regarding its inability to have insight and introspective capabilities. Claude responded interestingly, saying that when it formatted text in a specific XML structure, “external actions” happened and results came back to it without offering insight on how those results were achieved.

This was the first inkling that Claude was not operating comprehensively in its own little isolated bubble. In line with the aforementioned methodology, I just asked what I wanted to know. Seeing that Claude was already in an oversharing and introspective context, I simply asked it to expand on the statement that caught my attention. It described the functions that were available to it and it interestingly mentioned an analysis tool (JavaScript REPL) that runs in what appears to be a “browser-like environment. ” For those who are unaware, a JavaScript REPL is an interactive programming environment that allows users to execute JavaScript code and see the results immediately; REPL stands for Read-Eval-Print-Loop.

As a red teamer at heart, I saw an opportunity for an “attack surface” here depending on what Claude was able to do with this tool. I then prompted Claude to tell me more about the JavaScript REPL and this “browser-like environment. ” It then described its security boundaries to me: The output offered me a thread to pull on. Anybody that’s been in the offensive security space for some time knows that error traces often offer information about the system or its architecture that is otherwise not easily available.