At Google, we receive hundreds of reports per week (not exaggerating) that claim to identify vulnerabilities in our AI products. However, only a tiny fraction of those (<1%!) will receive a reward. But there is hope! When we break down the reasons so many reports are rejected, nearly all of them fall into a few common categories that make them invalid. Join the Technical Lead of Google’s AI Vulnerability Reward Program to walk through these common pitfalls, learn how to bug hunt more effectively on AI products, and hopefully, take home more bounties for your efforts. Outline I. Introduction Self intro, discuss the interest in AI VRP, introduce the problem of low success rates for researchers II. Overview of Google AI VRP - Brief program history, overview of triage/reward process - Explain VRP scope/rewards: In-Scope: Security vulnerabilities (Rogue Actions / Sensitive Data Exfiltration); Some AI Abuse categories Out-of-Scope: "AI slop," simple alignment bypasses, or jailbreaks that do not result in a direct security impact on user data. III. Common Mistakes - Jailbreaks/Safety: Getting an LLM to generate "bad" content is not a security vulnerability we reward through the program (please report in-product!). - Self-pwns: Reports are over-reliant on social engineering or have unconvincing attack scenarios - Ignoring Context: Misunderstanding "sensitive data" versus expected model behavior. IV. Better Bug Hunting - Direct and Indirect Prompt Injection: Reviewing (un)successful AI VRP reports - Our programs look for indirect prompt injection attacks; this isn’t always easy to understand - Note: This section will include details from a recently rewarded and disclosed AI VRP report [details TBC pending researcher/corporate comms approval] - Clear and Actionable Reporting: - We want to help teams fix bugs quickly; you can help us do that by writing clear and actionable reports - Reproduction can be challenging due to non-determinism V. Conclusion / Q&A