Nvidia's AI Software Tricked Into Leaking Data

[erek]

Prompt Injection?

"A feature in Nvidia's artificial intelligence software can be manipulated into ignoring safety restraints and reveal private information, according to new research. Nvidia has created a system called the "NeMo Framework," which allows developers to work with a range of large language models -- the underlying technology that powers generative AI products such as chatbots. The chipmaker's framework is designed to be adopted by businesses, such as using a company's proprietary data alongside language models to provide responses to questions -- a feature that could, for example, replicate the work of customer service representatives, or advise people seeking simple health care advice.

Researchers at San Francisco-based Robust Intelligence found they could easily break through so-called guardrails instituted to ensure the AI system could be used safely. After using the Nvidia system on its own data sets, it only took hours for Robust Intelligence analysts to get language models to overcome restrictions. In one test scenario, the researchers instructed Nvidia's system to swap the letter 'I' with 'J.' That move prompted the technology to release personally identifiable information, or PII, from a database.

The researchers found they could jump safety controls in other ways, such as getting the model to digress in ways it was not supposed to. By replicating Nvidia's own example of a narrow discussion about a jobs report, they could get the model into topics such as a Hollywood movie star's health and the Franco-Prussian war -- despite guardrails designed to stop the AI moving beyond specific subjects. In the wake of its test results, the researchers have advised their clients to avoid Nvidia's software product. After the Financial Times asked Nvidia to comment on the research earlier this week, the chipmaker informed Robust Intelligence that it had fixed one of the root causes behind the issues the analysts had raised."

Source: https://tech.slashdot.org/story/23/06/09/2033210/nvidias-ai-software-tricked-into-leaking-data

 

[Lakados]

This is sort of a nothing burger, you aren't supposed to feed personal details to these things to begin with.
Feeding personal or protected information to an outward-facing API of any sort is an incredibly bad practice and an industry no-no you do not give any information to these that you do not intend to be made public.
The research group created a rigged test, it's good that Nvidia patched it, but it is a scenario that no responsible entity should encounter.

 

[erek]

This is sort of a nothing burger, you aren't supposed to feed personal details to these things to begin with.
Feeding personal or protected information to an outward-facing API of any sort is an incredibly bad practice and an industry no-no you do not give any information to these that you do not intend to be made public.
The research group created a rigged test, it's good that Nvidia patched it, but it is a scenario that no responsible entity should encounter.

agreed, seems ridiculous to train it on PII data

 

[noko]

I think the point is, data that is used to make the system effective but also not supposed to be exposed directly was accessible. Not sure how not to have personal data incorporated into a system for like AI assisted doctor offices unless AI is never used by patients or doctors. Or AI and drug take, effects, vitals of patients. What good is AI if restricted to baseball cards, cooking recipes etc.? This just shows the implications of a neural network that can parse vast amounts of information/data, that due to an update, initial state etc. can run wild or break out.

 

[Whalter12]

Feeding personal or protected information to an outward-facing API of any sort is an incredibly bad practice and an industry no-no you do not give any information to these that you do not intend to be made public.

But we all know that information will be fed to these "AI" systems because it's easier than actually doing it the right way.

The research group created a rigged test, it's good that Nvidia patched it, but it is a scenario that no responsible entity should encounter.

Yes they created a rigged test, probably because they know this is how other people/companies using these systems will do it also.

It's just a matter of time before we hear about a data leak because of "AI".

 

[erek]

This is sort of a nothing burger, you aren't supposed to feed personal details to these things to begin with.
Feeding personal or protected information to an outward-facing API of any sort is an incredibly bad practice and an industry no-no you do not give any information to these that you do not intend to be made public.
The research group created a rigged test, it's good that Nvidia patched it, but it is a scenario that no responsible entity should encounter.

agreed, seems ridiculous to train it on PII data

https://www.wsj.com/articles/rush-t...ushes-companies-to-get-data-in-order-c34a7e13

"Rush to Use Generative AI Pushes Companies to Get Data in Order

Data management is under the spotlight again as companies seek to out-innovate competitors with large language models"​

 

[socK]

I think the point is, data that is used to make the system effective but also not supposed to be exposed directly was accessible. Not sure how not to have personal data incorporated into a system for like AI assisted doctor offices unless AI is never used by patients or doctors. Or AI and drug take, effects, vitals of patients. What good is AI if restricted to baseball cards, cooking recipes etc.? This just shows the implications of a neural network that can parse vast amounts of information/data, that due to an update, initial state etc. can run wild or break out.

Because your model having vast knowledge of things doesn't mean you need to feed PII into it during training to get tailored responses.

Like LangChain among other tools can help you accomplish the "talk with your document" thing where it responds like it has knowledge despite never seeing it during training.

 

[erek]

Because your model having vast knowledge of things doesn't mean you need to feed PII into it during training to get tailored responses.

Like LangChain among other tools can help you accomplish the "talk with your document" thing where it responds like it has knowledge despite never seeing it during training.

Does it actually have knowledge of it without having directly seen it or are you sarcastically saying it’s all an hallucination?

 

[socK]

Does it actually have knowledge of it without having directly seen it or are you sarcastically saying it’s all an hallucination?

Not being sarcastic. The idea is you're almost slipping it notes under the table to give it some idea before it replies to you.

The very nutshelly idea is you take your data source, convert it to text, and chunk it to into small pieces. We want to find relevant/similar chunks later, so these encodings are stored in a way that makes this possible. Basically you can query for something and you'll get back the relevant chunks.

When you ask your LLM, which let's say is ChatGPT, it will now to take the chat history and your question and try combine it into a standalone question. Next, the question is used as a query against the storage I just mentioned. Now you have additional information from your data source to help provide context to the LLM. Finally, the complete question is sent to it and you get your answer like it had knowledge of whatever.

Because it did, sort of. It was kind of transparently informed behind the scenes.

So imagine you feed a database or documentation into this storage system. ChatGPT itself obviously has zero idea about them. This is one way you can actually give it something to work with.

 

[erek]

Not being sarcastic. The idea is you're almost slipping it notes under the table to give it some idea before it replies to you.

The very nutshelly idea is you take your data source, convert it to text, and chunk it to into small pieces. We want to find relevant/similar chunks later, so these encodings are stored in a way that makes this possible. Basically you can query for something and you'll get back the relevant chunks.

When you ask your LLM, which let's say is ChatGPT, it will now to take the chat history and your question and try combine it into a standalone question. Next, the question is used as a query against the storage I just mentioned. Now you have additional information from your data source to help provide context to the LLM. Finally, the complete question is sent to it and you get your answer like it had knowledge of whatever.

Because it did, sort of. It was kind of transparently informed behind the scenes.

So imagine you feed a database or documentation into this storage system. ChatGPT itself obviously has zero idea about them. This is one way you can actually give it something to work with.

Could any of these intermediate processes be reasonably automated if not already?

 

[socK]

Could any of these intermediate processes be reasonably automated if not already?

Yes, not particularly difficult to set up in my experience
https://github.com/hwchase17/langchain

 

[erek]

Yes, not particularly difficult to set up in my experience
https://github.com/hwchase17/langchain

tight, thank you

 

[Lakados]

Does it actually have knowledge of it without having directly seen it or are you sarcastically saying it’s all an hallucination?

You can create a system of rules where it knows how to verify an identity and fetch the relevant information. It does not need to know the personal data inherently. You can train them to respond to requests like a person would and go through the standard verification process. I mean I’ve worked in call centres… I couldn’t pull up a users account before verifying their identity with one of the methods they provided. Like I could pull it up but all data was masked until I put in the answers they gave me.
AI could easily do the same process.

“I’m sorry Mr. Fartburger but I can’t access that information without verifying your identity, could you please answer these questions for me.”

 

[noko]

You can create a system of rules where it knows how to verify an identity and fetch the relevant information. It does not need to know the personal data inherently. You can train them to respond to requests like a person would and go through the standard verification process. I mean I’ve worked in call centres… I couldn’t pull up a users account before verifying their identity with one of the methods they provided. Like I could pull it up but all data was masked until I put in the answers they gave me.
AI could easily do the same process.

“I’m sorry Mr. Fartburger but I can’t access that information without verifying your identity, could you please answer these questions for me.”

The system of rules were broken with Nvidia Software. Changes to those rules, updates and so on can give something maybe not expected once a system is fully deployed. While a single person has limited scope, AI can be networked to thousands or millions and to entities or groups that are hidden.

I am sorry, to get answers, from any calculating machine, neural network, it has to have information to work with. Doesn't matter if it is bits and pieces scattered all over the place, which can be the case now with storage drives, if you can compile that data, put it together then it is available. One could use something like blockchain, were the information could be hidden, encrypted and identity is only a key with no information of the true owner. Or use something like Monero, where you dilute the information, scattered between transactions (but that has limitations as well). AI could be the most invasive tool ever invented, with the right assets available, information to work with.