BlackHat 2024: Copilot, Copilot, and more Copilot

It was quite the week in Las Vegas for hacker summer camp, where we had a great opportunity to meet with customers, partners, and security leaders to discuss the latest trends in hacking, cybersecurity, and of course… AI. Let’s dive in for our BlackHat in review.

In keeping with recent history, AI dominated the week, and we noticed that it is more or less at the top of every security practitioner’s list of priorities. However there is skepticism, as so many vendors are talking about AI, when in reality these capabilities are basically dressed-up automations. What CISOs and security teams are really looking for is a way to harness the power of AI, and get it in the hands of more and more of their business users.

Unfortunately, and as we’ll go into more detail, AI apps and copilots are juicy targets for attackers due to their sheer power, access to data, and ability to operate in the shadows due to the ‘black-boxiness’ of these AI providers and LLMs.

When done right, BlackHat and DefCon are great showcases to put security innovation at the forefront, and help security teams act as drivers for the business; not inhibitors.

The Risks From Internet Facing Copilots

As we’ve seen, AI continues to eat the world, and these conferences are often microcosms of this. At BlackHat, our CTO and Co-Founder, Michael Bargury was finally able to unveil some truly breathtaking security research that he and his team have been spearheading to show just how far we still have to go in order to keep AI from leaking sensitive corporate data. In his first talk, 15 Ways to Break Your Copilot, Michael showcased CopilotHunter, a tool that scans for any copilot that is publicly accessible to the internet, and then uses fuzzing and GenAI to abuse those public copilots to extract sensitive enterprise data.

The reason this is such a danger, and why it was such a well attended session, is that there are already tens of thousands of organizations leveraging Microsoft Copilot Studio, which is a low-code tool that allows anyone to extend Microsoft Copilot (more on this in a second) and/or build their own AI apps and copilots. As business users of all technical backgrounds are building their own copilots, one of the most common mistakes they’ll make is exposing one (or many) of their AI apps or copilots to the internet where anyone can use it. In some cases, this is totally fine; think of a copilot that someone might build to help users navigate a customer-facing website, but these public facing copilots cannot be linked or connected to sensitive internal datasets, or rest assured, a hacker will find a way to extract this data.

The talk showed that there are countless copilots that are exposed to the internet that shouldn’t be, with many more surely on the way as more and more users leverage this type of technology to drive business-led innovation and productivity. Maintaining inventory of copilots is key, but then taking it a step further to assess which of these are also connected with sensitive corporate data can be the difference between ending up in the crosshairs of CopilotHunter, or not.

The Big Kahuna: Hacking Microsoft Copilot

On Thursday, Michael unveiled another talk, where he showed how easy it is for attackers to live off the land of Microsoft Copilot, performing sweeping remote copilot execution to enable a bad actor to take control over the entirety of Microsoft Copilot.

From a hacker’s perspective, Copilot is a goldmine. As currently constructed, it is tailormade for attackers.

• Access to sensitive data? Check
• Ability to find that data easily? Check
• Exfiltrate data without a trace? Check
• Work on your behalf to help you move laterally? Check

In his talk, “Living Off Microsoft Copilot,” Michael analyzed Copilot from a red-team perspective, showing how plugins can be used to install backdoors into other users’ interactions, allowing for basic data theft, but then using AI on itself to social engineer other users to circumvent data classification and least privilege controls. Unfortunately, these are all capabilities available to hackers, and they can use basic prompts like emails and Teams messages to achieve these goals, as described in the talk.

After the talk (and leading into it as well for anyone who read Michael’s briefing description), there was a LOT of interest in ensuring that security and governance controls are in place to prevent these types of attacks, which represent essentially AI equivalents of SQL injection and remote code execution; something that’s being referred to as ‘remote copilot execution.’

Looking Ahead

While the threats looming are always massive, BlackHat and DefCon gives us hope that the good guys will prevail in the end. There are so many talented and brilliant security researchers and leaders, and it really is true that security is a team game. We are all in this together to thwart attacks, and the kernels for these defenses are research and breaking things. We loved learning and hearing from old friends and new about what they are doing to continue enabling the business to capitalize on all the latest technologies, while empowering security practitioners to keep up. Already looking forward to returning next year!