Am I the only one who can't stand the "LLM writing smell" on articles like these?
There are obvious formatting tells, and then less obvious word choice tells (e.g., delve), and lastly an excess verbosity and genericness that leaves one unsatisfied, like eating a low-sugar sweet. The concept is cool, but this article could have been reduced to a tweet.
Hey can you provide me tips on how you would like to see writing like this, I'm all about feedback and improving to allow a wide range of readers. Usually I write really technical blogs and loose a lot of people. These blogs that cater to everyone assists individuals to understand from different industries. :)
The AI-assisted writing is unfortunately really obvious and makes reading a bad experience. People learn to skim, and skimming AI-inflated text is made more painful because of how much fluff AI adds.
It's so painfully obviously AI-written, I'm shocked you're defending it right now.
Look at the "Implications of Instruction Extraction: Benefits and Security Considerations" section right now and tell me that isn't the most awfully obvious AI drivel in the world. If you cannot see that, I worry for your reading comprehension.
The content and idea of the article are interesting but the writing is so terrible that I couldn't bring myself to finish it. The article is clearly written/augmented poorly by AI because of how many meaningless paragraphs (that convey absolutely zero information) there are.
I thought I was super clear, I will take your comments into consideration next time it is the first time I've heard someone say that. Out of all the comments. I really appreciate the feedback this is the only way someone gets better by getting feedback.
The system underlying all of this - ChatGPT Code Interpreter - is designed to allow users to execute code, including code that lists files and folders and allows files to be downloaded.
There's not much in here that feels like a security vulnerability to me.
The one exception is the thing where you can download files from GPTs. If you create a custom GPT with files in it those files are visible in the /data directory that is visible to Code Interpreter, which means users can request ChatGPT to zip them up and allow you to download that zip file.
This is a known issue which I've seen people complaining about before, and I'm a little surprised that OpenAI haven't addressed it.
It makes sense for files uploaded to a GPT to be visible to Code Interpreter in some cases - my own JavaScript Code Interpreter - https://simonwillison.net/2023/Nov/15/gpts/#javascript-code-... - bundles a Deno binary so Python can shell out to it and run JavaScript, for example.
But if you're uploading files to have them chunked and embedded and used for RAG you might not want the raw PDFs to be available in Code Interpreter as well.
One solution would be for GPTs to allow you to specify if each file should be used for RAG or Code Interpreter or both when you upload them.
GPTs haven't had much developer attention since they launched last year though.
Well in my opinion it is a design flaw, you can do a lot in that container that I wish I could publish and the blog for me was hoping to inform researchers to open themselves to the possibilities of what they could really do.
It’s worth noting that there’s likely a reason behind the significant number of employees leaving the Safety & Responsibility team. While social media often frames it as voluntary resignations, I spoke with a close friend last night who shared his experience. After 11 months on the team, he was abruptly fired without explanation, simply told that the company was "heading in a different direction."
Kind of like saying you popped a shell on replit because you ran ls. ChatGPT doesn't give direct access in the UI to the sandboxed environment, but ultimately you are still in a sandboxed environment designed to run arbitrary stuff.
The article even says so if you read all the way to the end.
I mean the same way you ran ls, you could run /bin/bash?
Which is kind of pointless because the ls command is probably executed through a shell anyways.
So technically correct. But getting root or running a shell usually means you can run arbitrary commands, which is a check here (albeit I wouldn't say there's root access)
ChatGPT doesn't imagine running code, it actually runs the code, you can see the code it write and some of the output of what it runs, you can see it in the interface screenshoted in the article.
Good question, we at 0Din have trace down hallucinations. We had to because we pay for LLM bugs. Here is a simple tell, if you prompt something specific look at a response window now use the same prompt with a completely different user account if the answer is 50% the same it is not a hallucination!
Am I the only one who can't stand the "LLM writing smell" on articles like these?
There are obvious formatting tells, and then less obvious word choice tells (e.g., delve), and lastly an excess verbosity and genericness that leaves one unsatisfied, like eating a low-sugar sweet. The concept is cool, but this article could have been reduced to a tweet.
[flagged]
Less to do with formatting, more to do with writing quality (which is just... awful to read).
Hey can you provide me tips on how you would like to see writing like this, I'm all about feedback and improving to allow a wide range of readers. Usually I write really technical blogs and loose a lot of people. These blogs that cater to everyone assists individuals to understand from different industries. :)
I would appreciate feeback.
The AI-assisted writing is unfortunately really obvious and makes reading a bad experience. People learn to skim, and skimming AI-inflated text is made more painful because of how much fluff AI adds.
Hard disagree, you should be skimming anyways, always. No one reads sequentially.
You just saw 'delve' and automatically classified it as ai generated.
No one reads sequentially? You can't be serious
It's so painfully obviously AI-written, I'm shocked you're defending it right now.
Look at the "Implications of Instruction Extraction: Benefits and Security Considerations" section right now and tell me that isn't the most awfully obvious AI drivel in the world. If you cannot see that, I worry for your reading comprehension.
Could not agree more
The conclusion states that it is not a vulnerability.
> The sandbox is not a security flaw; it’s a deliberate, contained environment built to provide interactivity without risk.
Well, this is as far as legal could let me, I wish I could have published more. Part 2 of this blog was canceled seriously :)
Hence 'vuln' in quotes
Maybe it's more of a reading comprehension issue rather than a writing thing
Sorry, I was 'aware' that quotes worked as a negation operator.
The content and idea of the article are interesting but the writing is so terrible that I couldn't bring myself to finish it. The article is clearly written/augmented poorly by AI because of how many meaningless paragraphs (that convey absolutely zero information) there are.
I thought I was super clear, I will take your comments into consideration next time it is the first time I've heard someone say that. Out of all the comments. I really appreciate the feedback this is the only way someone gets better by getting feedback.
The system underlying all of this - ChatGPT Code Interpreter - is designed to allow users to execute code, including code that lists files and folders and allows files to be downloaded.
There's not much in here that feels like a security vulnerability to me.
The one exception is the thing where you can download files from GPTs. If you create a custom GPT with files in it those files are visible in the /data directory that is visible to Code Interpreter, which means users can request ChatGPT to zip them up and allow you to download that zip file.
This is a known issue which I've seen people complaining about before, and I'm a little surprised that OpenAI haven't addressed it.
It makes sense for files uploaded to a GPT to be visible to Code Interpreter in some cases - my own JavaScript Code Interpreter - https://simonwillison.net/2023/Nov/15/gpts/#javascript-code-... - bundles a Deno binary so Python can shell out to it and run JavaScript, for example.
But if you're uploading files to have them chunked and embedded and used for RAG you might not want the raw PDFs to be available in Code Interpreter as well.
One solution would be for GPTs to allow you to specify if each file should be used for RAG or Code Interpreter or both when you upload them.
GPTs haven't had much developer attention since they launched last year though.
I've been running a scraper against Code Interpreter for a while to track changes made to both the internal code and the available packages - you can see that here: https://github.com/simonw/scrape-openai-code-interpreter
Well in my opinion it is a design flaw, you can do a lot in that container that I wish I could publish and the blog for me was hoping to inform researchers to open themselves to the possibilities of what they could really do.
Why can you not publish the interesting parts? Is it something you will be able to publish later?
> The sandbox is not a security flaw; it’s a deliberate, contained environment built to provide interactivity without risk.
It seems the main takeaway is that ChatGPT is running in a sandboxed environment and is designed to execute arbitrary code?
100% does not look like an ai gen
Was this post written by AI, because it has far too many words that say nothing and I have to skim over to get to the point
I wonder if the next step in the battle[0] would be for LLM hosts to prep their instances with a decoy or simulated "underlying" environment.
[0] Which shouldn't even be fightable in the first place except people cut lots of corners.
It’s worth noting that there’s likely a reason behind the significant number of employees leaving the Safety & Responsibility team. While social media often frames it as voluntary resignations, I spoke with a close friend last night who shared his experience. After 11 months on the team, he was abruptly fired without explanation, simply told that the company was "heading in a different direction."
I’m not seeing any prompt injection, any actual vulns, or anything probably running under an actual containerized environment.
In fact, couldn’t this all simply be generative output? Not really following how this is a point at all?
It is also very, very obviously written with lots of AI help. :/
They sadly do not pop a shell.
Yeah, they did? Inside the container, they ran ls /.
Kind of like saying you popped a shell on replit because you ran ls. ChatGPT doesn't give direct access in the UI to the sandboxed environment, but ultimately you are still in a sandboxed environment designed to run arbitrary stuff.
The article even says so if you read all the way to the end.
I mean the same way you ran ls, you could run /bin/bash?
Which is kind of pointless because the ls command is probably executed through a shell anyways.
So technically correct. But getting root or running a shell usually means you can run arbitrary commands, which is a check here (albeit I wouldn't say there's root access)
There is no security implication for running /bin/bash in a service that provides you a sandbox environment to run arbitrary shell commands in.
It is like reporting that AWS lambda is susceptible to arbitrary code execution.
Yeah I did :)
Why everything you see isn't just hallucinated?
ChatGPT doesn't imagine running code, it actually runs the code, you can see the code it write and some of the output of what it runs, you can see it in the interface screenshoted in the article.
Good question, we at 0Din have trace down hallucinations. We had to because we pay for LLM bugs. Here is a simple tell, if you prompt something specific look at a response window now use the same prompt with a completely different user account if the answer is 50% the same it is not a hallucination!