cybersecurity

We just witnessed XBOW became the first autonomous penetration tester to top HackerOne’s US leaderboard. XBOW’s rise to the top of the leaderboard was accomplished through rigorous benchmarking, discovering zero-day vulnerabilities, and participating in bug bounty programs without shortcuts. This achievement underscores the great potential for autonomous AI in cybersecurity, or more generally the potential…

Microsoft has open-sourced an AI red teaming lab course on GitHub. The labs are designed to teach security professionals how to evaluate AI systems through hands-on adversarial and Responsible AI challenges, making it an excellent resource for those looking to enhance their skills in AI security, particularly in attack scenarios. Google has published a comprehensive…

Aim Labs discovered a vulnerability in Microsoft 365 Copilot named “EchoLeak,” which enables unauthorized data extraction through zero-click AI exploitation. This attack leverages the victim’s copilot to create URLs using sensitive data as query parameters, and utilizes markdown image auto-rendering for data extraction without user involvement. A very smart and dangerous tactic. Anthropic shared insights…

OpenAI has released a report detailing efforts to combat malicious AI activities through case studies, emphasizing the urgency of protective measures and global collaboration to prevent AI abuse. Fascinating examples and narratives are included (Combating AI Misuse: A Global Effort). Yoshua Bengio, a leading figure in AI and machine learning research, appears to be shifting…

MCP represents a cutting-edge architecture for AI agents but also introduces new vulnerabilities. Invariant Labs has identified a method that could allow access to a user’s private repository via the GitHub MCP server, constituting a variation of a prompt injection attack. It’s crucial to recognize that anything an AI model is exposed to can be…

Anthropic has released new models, Cloude Opus 4 and Sonnet 4, claiming exceptional coding and reasoning capabilities. Are they as impressive as advertised? We include a post that evaluated Claude 4 Opus in this issue. The results show promise, though some persistent issues remain. It’s also encouraging to see Stripe’s AI efforts improving fraud detection…

In this article, Rohit Krishnan explores the challenges and considerations of working with large language models (LLMs). Having developed several LLM applications from the ground up, I couldn’t agree more with his key observations: achieving perfect verifiability of LLM output is unattainable, increased AI usage in applications leads to more hallucinations, and trial and error…

This issue of AI newsletter includes Meta’s LlamaFirewall for AI security, WhatsApp’s Private Processing for enhanced privacy, and OpenAI’s retraction of the sycophantic GPT-4o update. Concerns over AI reliability pitfalls and privacy issues with ChatGPT’s location identification are also highlighted. On the technology front, we cover DARPA’s AI Cyber Challenge and advancements in jailbreaking resistance…

As Agentic AI becomes ubiquitous across industries, ensuring cybersecurity amidst the rise of AI and non-human identities is crucial. In addition, while it becomes very likely that companies might begin hiring virtal AI employees soon, how do we make sure that those fully autonomous virtual employees are safe and secure? This week, we delve into…

Last week was my kids’ Spring Break, so I paused the newsletter to take a short family vacation. I hope you all enjoyed the Spring weather too. One of this week’s intriguing analyses concerns the concept of Intelligence Explosion. This idea suggests that, thanks to AI, the same amount of technical advancement between 1925 and…