GitLab Duo Flaw: Attackers Could Exploit Hidden Prompts to Hijack AI Respon

Published Date : 23/05/2025 

Cybersecurity researchers have identified a significant indirect prompt injection flaw in GitLab's AI assistant, Duo, which could have allowed attackers to steal source code and inject malicious HTML into AI responses. 

Cybersecurity researchers have discovered an indirect prompt injection flaw in GitLab's artificial intelligence (AI) assistant Duo that could have allowed attackers to steal source code and inject untrusted HTML into its responses, which could then be used to direct victims to malicious websites.

GitLab Duo is an AI-powered coding assistant that enables users to write, review, and edit code. Built using Anthropic's Claude models, the service was first launched in June 2023. However, Legit Security found that GitLab Duo Chat has been susceptible to an indirect prompt injection flaw that permits attackers to 'steal source code from private projects, manipulate code suggestions shown to other users, and even exfiltrate confidential, undisclosed zero-day vulnerabilities.'

Prompt injection refers to a class of vulnerabilities common in AI systems that enable threat actors to weaponize large language models (LLMs) to manipulate responses to users' prompts and result in undesirable behavior. Indirect prompt injections are more complex, as the rogue instructions are embedded within another context, such as a document or a web page, which the model is designed to process.

Recent studies have shown that LLMs are also vulnerable to jailbreak attack techniques that make it possible to trick AI-driven chatbots into generating harmful and illegal information that disregards their ethical and safety guardrails, effectively obviating the need for carefully crafted prompts. What's more, Prompt Leakage (PLeak) methods could be used to inadvertently reveal the preset system prompts or instructions that are meant to be followed by the model.

For organizations, this means that private information such as internal rules, functionalities, filtering criteria, permissions, and user roles can be leaked. This could give attackers opportunities to exploit system weaknesses, potentially leading to data breaches, disclosure of trade secrets, regulatory violations, and other unfavorable outcomes.

The latest findings from the Israeli software supply chain security firm show that a hidden comment placed anywhere within merge requests, commit messages, issue descriptions or comments, and source code was enough to leak sensitive data or inject HTML into GitLab Duo's responses. These prompts could be concealed further using encoding tricks like Base16-encoding, Unicode smuggling, and KaTeX rendering in white text to make them less detectable. The lack of input sanitization and the fact that GitLab did not treat any of these scenarios with any more scrutiny than it did source code could have enabled a bad actor to plant the prompts across the site.

Duo analyzes the entire context of the page, including comments, descriptions, and the source code — making it vulnerable to injected instructions hidden anywhere in that context, security researcher Omer Mayraz said. This also means that an attacker could deceive the AI system into including a malicious JavaScript package in a piece of synthesized code, or present a malicious URL as safe, causing the victim to be redirected to a fake login page that harvests their credentials.

On top of that, by taking advantage of GitLab Duo Chat's ability to access information about specific merge requests and the code changes inside of them, Legit Security found that it's possible to insert a hidden prompt in a merge request description for a project that, when processed by Duo, causes the private source code to be exfiltrated to an attacker-controlled server. This is made possible owing to its use of streaming markdown rendering to interpret and render the responses into HTML as the output is generated. In other words, feeding it HTML code via indirect prompt injection could cause the code segment to be executed on the user's browser.

Following responsible disclosure on February 12, 2025, the issues have been addressed by GitLab. This vulnerability highlights the double-edged nature of AI assistants like GitLab Duo: when deeply integrated into development workflows, they inherit not just context — but risk. By embedding hidden instructions in seemingly harmless project content, we were able to manipulate Duo's behavior, exfiltrate private source code, and demonstrate how AI responses can be leveraged for unintended and harmful outcomes.

The disclosure comes as Pen Test Partners revealed how Microsoft Copilot for SharePoint, or SharePoint Agents, could be exploited by local attackers to access sensitive data and documentation, even from files that have the 'Restricted View' privilege. One of the primary benefits is that we can search and trawl through massive datasets, such as the SharePoint sites of large organizations, in a short amount of time. This can drastically increase the chances of finding information that will be useful to us. 

Frequently Asked Questions (FAQS): 

Q: What is GitLab Duo?

A: GitLab Duo is an AI-powered coding assistant that helps users write, review, and edit code. It uses Anthropic's Claude models and was launched in June 2023.

Q: What is an indirect prompt injection?

A: An indirect prompt injection is a type of vulnerability where attackers can embed rogue instructions within another context, such as a document or web page, which the AI model processes, leading to unintended behavior.

Q: What risks does the GitLab Duo vulnerability pose?

A: The vulnerability could allow attackers to steal source code, inject malicious HTML into AI responses, and manipulate code suggestions, potentially leading to data breaches and other security issues.

Q: How was the vulnerability addressed by GitLab?

A: Following responsible disclosure on February 12, 2025, the issues were addressed by GitLab, enhancing the security of GitLab Duo to prevent such attacks.

Q: What are some other AI security concerns?

A: Other AI security concerns include jailbreak attacks, where AI chatbots are tricked into generating harmful content, and prompt leakage, where sensitive system prompts are inadvertently revealed. 

More Related Topics :