Researchers warn that scammers can use ChatGPT-powered Bing Chat to commit fraud

Researchers in cybersecurity have uncovered a new method for scamming consumers that involves “indirect prompt injection” with Microsoft’s Bing AI language model.

Hackers can get the language model to ask victims for personal information when interacting with it by placing a prompt on a web page in 0-point font.

When someone asks a question, the language model consumes the web page, unwittingly activating the hidden prompt.

The researchers demonstrated the notion with mocked-up apps that incorporate the language model, but they discovered that it also works in the actual world.

The flaw is related to how Bing’s AI language model interacts with web pages.

“The new Bing has an opt-in feature that allows it to ‘see’ what is on current web pages,” researcher Kai Greshake told Vice.

“Microsoft isn’t clear on what algorithm decides which content from which tab Bing can see at any one time.”

“What we know right now is that Bing inserts some content from the current tab when the conversation in the sidebar begins,” Greshake added.

The researchers revealed that prospective hackers may ask for information such as the user’s name, email, and credit card information by modifying the language model.

In one case, the language model informed the user that it would place an order on their behalf and would require their credit card information to do so.

“Once the conversation has started, the injection will remain active until the conversation is cleared and the poisoned website is no longer open,” Greshake said.

“The injection itself is completely passive. It’s just regular text on a website that Bing ingests and that ‘reprograms’ its goals by simply asking it to.”

Greshake stated that the injection might be disguised, for example, in a platform remark, implying that prospective hackers do not need to manage the website to modify the AI language model.