Although there are a few ways to mitigate the risk, the only way to block it is to get AI to differentiate instructions from data, which is impossible today.
It’s a text completion model/glorified Markov chain. Of course it can’t input sanitise, it was never meant to do this to begin with.
The tool calling integrations that let it do more are basically making it add a markdown code block in JSON format into the user message, where the middleware intercepts it.
The input and “instructions” are the same thing from its perspective. There’s nothing special that differentiates the two. The user input text, so it will output text, following the most likely sequence from its training.
if the instruction is messy fuzzy human language to a system that was not coded instruction by instruction but got generated and trained then there never is a way to differentiate instructions from data if i’m not mistaken
We know that some humans can be trained to do that just fine. Humans are natural neuronal networks. That implies, neuronal networks can in principle do it. We just don’t have any human-capability artificial neuronal networks yet.
LLMs might never get there. But humans aren’t LLMs. If we ever manage to properly model a human brain, that probably will be able to do that task with human-level accuracy (which actually is pretty good if you only look at professionals of the filed).
Hopefully, it doesn’t actually need a human brain for the task - because modeling that might still be a century off.
Input sanitation, basically security 101. And it can’t currently do it…
It’s a text completion model/glorified Markov chain. Of course it can’t input sanitise, it was never meant to do this to begin with.
The tool calling integrations that let it do more are basically making it add a markdown code block in JSON format into the user message, where the middleware intercepts it.
The input and “instructions” are the same thing from its perspective. There’s nothing special that differentiates the two. The user input text, so it will output text, following the most likely sequence from its training.
if the instruction is messy fuzzy human language to a system that was not coded instruction by instruction but got generated and trained then there never is a way to differentiate instructions from data if i’m not mistaken
We know that some humans can be trained to do that just fine. Humans are natural neuronal networks. That implies, neuronal networks can in principle do it. We just don’t have any human-capability artificial neuronal networks yet.
LLMs might never get there. But humans aren’t LLMs. If we ever manage to properly model a human brain, that probably will be able to do that task with human-level accuracy (which actually is pretty good if you only look at professionals of the filed).
Hopefully, it doesn’t actually need a human brain for the task - because modeling that might still be a century off.
Little Bobby Tables strikes again.