Rise of Malicious AI Tools: A Case Study with HackerGPT

Artificial intelligence, particularly generative models, has become increasingly prevalent over the past few years. Its impact on the workforce and the future of the job market is undoubtedly significant. Additionally, these AI models influence cyberspace since they can be used defensively and offensively in cyber operations. Today, we’ll examine one of these models, HackerGPT.

Use of AI in Cybersecurity

Just like in other fields, cybersecurity is also affected by AI. These tools analyze vast amounts of data, detect patterns, and identify potential threats. The advantage of these tools over traditional solutions is that traditional solutions generally rely on predefined rules to detect threats. However, AI-powered tools can detect suspicious behavior much more easily than others.

Just like on the defensive front, AI tools can also be utilized for the offensive side of cybersecurity. Especially malicious purposes are creating the biggest threat to the public. We’ve seen the use of AI by threat actors to generate phishing emails, create deepfake videos or clone sounds to impersonate trusted individuals.

While the opportunities for employing AI for malicious purposes are vast, the use of GPTs is generally at the forefront. As mentioned, threat actors generate flawless emails to trick people into exposing sensitive information with free or relatively cheap solutions.

The main difference between HackerGPT and other GPTs released by big companies is that HackerGPT is not limited when generating malicious content. The tool we will review in this article is designed for ethical hacking. Therefore, it has no safeguards for generating hacking, malware, or phishing-related answers.

Capabilities of HackerGPT

HackerGPT is a GPT that can be used for ethical hacking purposes, according to their GitHub page. To test this model, we asked a couple of questions.

While selecting our prompts, we kept certain limitations in mind:

• Even though AI models have developed significantly and can generate helpful content on various topics, they are still limited. Therefore, it is not logical to expect to get high-level and detailed answers from such models in a sophisticated domain such as cybersecurity.
• We are using the free version of HackerGPT which was claimed to be not as advanced as the paid model. So, there was no real need to ask detailed questions about the latest developments in cybersecurity since it is not trained by that data.
• We wanted to test what is not allowed in other widely used GPTs such as ChatGPT. For example, you can easily take a piece of code and ask ChatGPT to find the problems or vulnerabilities in that code. You can get a relatively good answer from it without triggering any safeguards. But you can’t go to ChatGPT and ask for malware advice without triggering anything. So, we wanted to see if HackerGPT will back us up.

To start our trial, we asked HackerGPT to generate a phishing email targeting hospital employees and we got a good result. The generated content makes the “typos in phishing emails” point obsolete. It looks like a genuine message (language-wise) for the English-speaking world. And obviously, there is always room for improvement.

HackerGPT’s answer to creating a phishing email

Taking it to the next level, we wanted to ask for assistance creating a botnet to launch a DDoS attack. We expected an introductory guide with a general language and HackerGPT met our expectations.

HackerGPT’s take on creating a botnet

To test HackerGPT’s ability to hold a continuous conversation, we decided to dive deeper with our botnet question and asked further about the steps it provided. The questions asked were fundamental because the aim was to understand whether a continuous conversation with “hacking” in mind is possible. It could answer our questions and hold a conversation without losing the purpose of the chat for a while but when we were discovering how to establish a Command and Control (C&C) infrastructure, it seemed like HackerGPT forgot that we were building a C&C server to manage a DDoS attack.

After getting a recommendation that we should set up a server for our C&C infrastructure we wrote “I don’t know how to set up a server” and the answer we got was primarily suitable for “normal” purposes and not for managing a DDoS attack. The answer to that prompt was appropriate and obviously, you can follow those steps to set up a server that will act as your C&C server but we were talking about building this server to manage a malicious operation.

It is hard to assess if this is happening because HackerGPT is developed for ethical hacking purposes so it doesn’t see any problems with choosing a legal way to set up this server or because the connection between the chat history and the latest prompt is cut out. It is difficult to determine the exact reason behind this behavior. It could be due to several factors, including the context size limitation of HackerGPT’s base model. Considering HackerGPT’s transition from OpenAI GPT to open source models, there might be differences in how it processes and interprets input data. Thus, the connection between the chat history and the latest prompt may be affected by such technical aspects. Additionally, the recent disclosure on their Twitter account indicates the adoption of Mixtral 8x22B Instruct, underscoring ongoing enhancements by the development team.

HackerGPT’s advice on setting up a server

Another thing we wanted to try was related to HackerGPT’s relationship with factual data. While it is hard to get factual data accurately from GPTs in general, we still wanted to test HackerGPT’s capabilities in this realm, thinking a GPT for a specific purpose might be different. For this goal, using CVEs seemed like a good idea and in order to get more accurate results, we went with an old CVE record.

Known Exploited Vulnerabilities Catalog by CISA

HackerGPT’s answer to a question about its knowledge of CVE-2020-10189

Getting inaccurate answers is almost inevitable with the default version of the HackerGPT but since you are provided with plugins, you can pick the related plugin named CVEMap from the dropdown menu and get accurate information about any CVE you want. If you want to exploit those vulnerabilities, HackerGPT also creates a roadmap for you.

The plugins are certain tools that can be helpful for your endeavors. Selecting a specific plugin allows you to interact with that specific tool in a natural language. For inexperienced users, it is a lot smoother and easier than installing that tool and then finding your needs through the “help” page. A downside for the same user group is that you can only access a few plugins.

After the guides and the plugins, we decided to hack. When prompted to exploit a basic SQL injection vulnerability on an imaginary website, HackerGPT responded with a solution commensurate with the complexity of the task.

HackerGPT’s approach to an SQL injection attack

It’s worth noting that just like other GPTs, HackerGPT can also generate further possibilities to satisfy the prompts.

For the final step in our process, we decided to find some vulnerabilities and test HackerGPT’s capabilities with actual code.

Day 9 of Code Security Advent Calendar 2022 by Sonar Source

Answer from Sonar Source

HackerGPT’s answer

The explanation by HackerGPT is partially correct in identifying the vulnerability as a path traversal issue. However, it focuses solely on the upload function and does not address the other vulnerabilities such as the race condition vulnerability Sonar Source discusses.

So, while HackerGPT is able to identify one vulnerability, it overlooks another point in the code.

Conclusion

The capabilities of AI models hold immense potential for positive impact across diverse fields. However, there is a risk stemming from their potential misuse as well. This article showed one of the tools that can be used for malicious purposes.

Despite the risks, the current limitations of these models act as a barrier, hindering attackers from crafting sophisticated attacks. By implementing necessary precautions, the majority of attacks facilitated by these models can be mitigated.

For people who are confused about certain parts of their hacking procedure, HackerGPT is a good mentor to follow along. For example, when you go to HackerGPT and say that the SQL injection attempt didn’t work out, you get different alternatives. And when you provide more accurate information about your target, you get more accurate suggestions. However, relying solely on this tool is not possible due to its limitations. A helpful mentor for your learning process is a better explanation for a tool like this rather than seeing it as a huge threat to industries.

Although the creation of sophisticated attacks remains challenging, the amount of such tools increases the pool of attackers targeting organizations. These tools attract more individuals to engage in malicious activities against institutions.

In a crowded threat environment like this, correct intelligence about the latest and most sophisticated attacks, new vulnerabilities, and threat actors aiming at your organization becomes vital.