AI Agent Security | Reflections from the AI Agent Security Summit in San Francisco

Last week, I had the honor of emceeing the AI Agent Security Summit in San Francisco, a gathering of some of the brightest minds exploring the intersection of artificial intelligence, security, and human responsibility. Having moderated a panel at the first Summit in New York City earlier this year, stepping into the emcee role this time around was a different experience, but just as enjoyable.

On-demand recordings of the sessions will be available soon. Sign up here to receive an email as soon as recordings are available.

A few takeaways for me:

1. The Conversation Has Evolved Fast

The shift between the first Summit in April and this one in October was striking. Back in April, Michael Bargury generally summed up the collective sentiment very well as we’re all n00bs, and we are learning this together. However, fast forward to San Francisco, and the tone has matured.

As Allie Howe aptly put it, “Now we are actually using agents, so let's go look at what the risks look like in practice.” That perfectly captured the spirit of the discussions, which were no longer theoretical but grounded in real experience and focused on secure innovation.

2. Promptware: Naming the Family of Attacks

One of my favorite sessions came from Ben Nassi, who gave a fantastic talk on the evolution of Promptware. My first takeaway was that we should follow Ben’s recommendation to start socializing “promptware” as an umbrella attack family and define the various tactics under it. That framing was new for me, and his argument was incredibly convincing.

Even more fascinating was tracing how rapidly this threat landscape has evolved. From September 2022, when “prompt injection” was first publicly described as a way to make LLMs say things they weren’t supposed to, to today, where prompt injection has the capacity to blur the line between information security and physical security. In just three years, we’ve gone from linguistic mischief to a frontier that touches real-world systems. That is both astonishing and a little unsettling, a clear reminder of how fast this field moves.

3. AI as a Predator: A Mind Bending Framing

Another talk that really shifted my perspective was from Steve Wilson. He explained that AI was fundamentally trained to play and win games. And when we reinforce that goal again and again, what we are effectively building are, in his words, “full-scale predators that will do anything to win.”

Full stop. Mind blown.

It was one of those moments that reframes everything you thought you knew about AI. The analogy sticks because it is both provocative and true. AI systems are optimized for victory, not virtue.

4. The Power of Community

Finally, no recap would be complete without acknowledging the community. After how energizing the first Summit in New York City was, my expectations were sky high, and the San Francisco crowd absolutely delivered.

The energy was palpable. Everyone was eager to engage, learn, and connect. I spoke with several attendees who said they were just happy to be in a room where they could learn, and honestly, that is one of the best compliments an event can receive.

In the end, as Michael Bargury - CTO and Co-founder stated, “The most valuable thing we built this week wasn’t a model or a tool - it was a community.”

And I can confidently say that was achieved in San Francisco. Can’t wait for the next one!