Introduction

Hello! I'm Tanachu from the Security and Privacy Group.

In this article, I'll cover a recent security incident at our company involving a generative AI chat tool. Although the technical issue is not particularly new, I would like to introduce it because the circumstances under which it occurred were somewhat unusual.

Overview of the Case

Our company has created an environment where all employees can easily use the AI ​​chat tool. One day, while using the tool, an employee asked the AI a question. When they clicked on one of the links provided in the response, it led to a support scam website.

Image of a generative AI chat

Image of a support scam website

Even though we had security products in place, they couldn't block the messages, and the user was ultimately led to a support scam website. Fortunately, thanks to the employee's calm and quick judgment, we were able to avoid any serious damage.

This experience was a clear reminder that, just like with regular web searches, the links provided by generative AI can't always be trusted.

Cause of the Incident

To find out why the generative AI ended up suggesting a scam support website, we looked up the website it pointed to using the internet archive service WAYBACK MACHINE. We found that the website had actually provided what seemed to be legitimate content in the past.

We also discovered several other websites that listed this website as a reference.

This suggests that the generative AI may have learned from the past content of the website in question, along with information about the websites it had introduced. As a result, a hallucination-like phenomenon may have occurred, which led the AI to present incorrect information and possibly suggest a link to a scam support website.

Here's a summary of how the content on the problematic websites has changed, based on the investigation results.

• 2012 to early 2018 There was content history matching the domain name during this period. It is assumed that the site was considered reliable at this time.

• Late 2018 to early 2019 A domain management service sales page was shown, suggesting that the owner might have let go of the domain.

• Since late 2019 The domain has shown a history of unrelated content, such as medical issues, online casinos, fake warning screens, and domain parking.

For information on why security products were unable to detect the virus, check out "Appendix: Research Notes."

Measures for Similar Cases

At this point, it seems extremely difficult to take fundamental steps against cases like this, for the following reasons.

Challenges in Learning Generative AI

One possible reason behind this incident is that the generative AI may have learned from older content on the website in question, as well as from other websites it featured. Even if the content of a website is later updated, it's unlikely that those changes will be reflected in the AI's responses, including any related security risks.

The Best Prevention is Awareness

What we can take away from this case is that links suggested by generative AI aren't always safe. That's why it's so important to learn from real examples and understand how to respond if you actually encounter one.

Summary

This case was a somewhat unusual one, showing that links presented by generative AI are not necessarily safe. Depending on the characteristics of the fraudulent site, it may be difficult for security products to detect it. Also, if the site content is changed after the generative AI has learned, it may not be able to reflect the risk.

There may not be a solid solution at this point, but learning from cases like this can be a good reminder to stay alert about security when using generative AI.

Appendix: Research Notes

These are notes on investigating into why the issue wasn't picked up by security products. Please keep in mind this was a simple research, so it's just for reference.

1. Security Vendor Detection Status

We looked up the domain of the problematic website using our security tools and VirusTotal. Almost all the vendors flagged it as "safe."

2. Website Source Code

After checking the source code of the website in question, it seems there's a mechanism in place that sends a request to "domaincntrol[.]com" (note the missing o after the c) and then uses the response to dynamically decide where to direct the visitor.

We tried accessing the site a few times in a safe environment, and found that it redirected us to various other websites, including support scam pages and domain parking websites. This may have allowed the malware to slip past detection by security vendors.

3. Hosting Environment of Support Scam Website

The final support scam website is hosted on the "web.core.windows[.]net" domain, presumably utilizing the Microsoft Azure environment. Not just limited to Microsoft Azure, scam websites hosted on cloud services are generally hard to block with security products. That's because blocking cloud service domains outright isn't realistic, given how easy it is to set them up and how much it could impact legitimate business operations.

*As of this article's publication, we've confirmed that this support scam website has been taken down from Microsoft Azure.

4. Survey Results Using PublicWWW

Using a tool called PublicWWW, which lets you search website source code, we searched the distinctive string "domaincntrol[.]com/?orighost=" found on the problematic websites. The search revealed that this string appears in the code of over 20,000 websites. We also checked a few of those websites in more detail and found that they showed the same behavior — redirecting users to scammy support websites.