ASW #234 – Frank Catucci
Full Audio
View Show IndexSegments
1. Real-life Examples. Benefits, Risk & Security Implications of AI – Frank Catucci – ASW #234
With the increased interest and use of AI such as GTP 3/4, ChatGPT, GitHub Copilot, and internal modeling, there comes an array of use cases and examples for increased efficiency, but also inherent security risks that organizations should consider. In this talk, Invicti’s CTO & Head of Security Research Frank Catucci discusses potential use cases and talks through real-life examples of using AI in production environments. Frank delves into benefits, as well as security implications, touching on a number of security aspects to consider, including security from the supply chain perspective, SBOMs, licensing, as well as risk mitigation, and risk assessment. Frank also covers some of the types of attacks that might happen as a result of utilizing AI-generated code, like intellectual property leaking via a prompt injection attack, data poisoning, etc. And lastly, Frank shares the Invicti security team's real-life experience of utilizing AI, including early successes and failures.
Segment Resources:
- On-demand webinar on the topic of generative AI - https://www.scworld.com/cybercast/generative-ai-understanding-the-appsec-risks-and-how-dast-can-mitigate-them
- Invicti Research - https://www.invicti.com/blog/web-security/analyzing-security-github-copilot-suggestions/
- https://github.com/svenmorgenrothio/Prompt-Injection-Playground
This segment is sponsored by Invicti. Visit https://securityweekly.com/invicti to learn more about them!
Announcements
Security Weekly listeners save $100 on their RSA Conference 2023 Full Conference Pass! RSA Conference will take place April 24-27 in San Francisco and on demand. To register using our discount code, please visit https://securityweekly.com/rsac2023 and use the code 53UCYBER! We hope to see you there!
Guest
Frank Catucci is a global application security technical leader with over 20 years of experience, designing scalable application security specific architecture, partnering with cross-functional engineering and product teams. Frank is a past OWASP Chapter President and contributor to the OWASP bug bounty initiative and most recently was the Head of Application & Product Security at Data Robot. Prior to that role, Frank was the Sr. Director of Application Security & DevSecOps and Security Researcher at Gartner, and was also the Director of Application Security for Qualys. Outside of work and hacking things, Frank and his wife maintain a family farm. He is an avid outdoors fan and loves all types of fishing, boating, watersports, hiking, camping and especially dirt bikes and motorcycles.
Hosts
2. OpenAI Info Leak, BitCoin ATM Hack, GitHub RSA SSH Key, Measuring AI Security – ASW #234
Ferrari refuses ransomware, OpenAI deals with security issues from cacheing, video killed a crypto ATM, GitHub rotates their RSA SSH key, bypassing CloudTrail, terms and techniques for measuring AI security and safety
Announcements
Security Weekly listeners: Identiverse 2023 is heading to Vegas! Join the digital identity community at the ARIA Resort & Casino in Las Vegas, May 30th to June 2nd. Identiverse is a must-attend annual event that brings together over 2,500 security professionals for 4 days of world-class learning, engagement, and entertainment.
As a community member, you’re able to receive 20% off your Identiverse 2023 tickets using code IDV23-SW20!
Register today: securityweekly.com/identiverse2023
Hosts
- 1. Ferrari confirms extortion attempt, but car maker refuses to pay ransom
Nope, I don't own a Ferrari, but I grew up watching Thomas Magnum drive a Ferrari 308 GTS on Magnum, P.I. That's also pretty much the only reason I grabbed this article, other than to use it as a compare and contrast with OpenAI's notification this week.
Ferrari's response is light on details, but the article notes "...no payment details and/or bank account numbers or other sensitive payment information, nor details of Ferrari cars owned or ordered had been stolen." So, at least there's an implication that Ferrari had sufficient logs and monitoring to make such statements.
- 2. March 20 ChatGPT outage: Here’s what happened
The initial communication about this information leak was rough, with a "we feel awful about this" variation on taking security seriously and creeping towards a blame-the-intern approach by shifting the blame to an open source library.
This blog post makes up for that -- although I'd still quibble with the framing of "outage" as opposed to a clearer indication of account information leak. They go into details of the scope of who may have been affected, what what disclosed, what the underlying problem was, and how it was addressed. That kind of transparency is what we like to see in these kinds of write-ups.
The underlying problem was a race condition in redis-py, whose fix seems pretty easy to understand in hindsight. They used Python's "asyncio.shield()" method to protect the request/response queue from being corrupted by a cancelled request. Also nice to see tests added with this commit.
It also looks like OpenAI dealt with another cacheing issue that could lead to an account takeover. Bug bounty researcher @naglinagli describes the flaw in that thread, which includes some links to "Web Cache Deception Attacks" (also this pdf).
- 3. Zero-Day Bug Allows Crypto Hackers to Drain $1.6M From Bitcoin ATMs
I almost skipped over this because -- yawn -- another crypto hack. But in reading the security incident, I was struck by the three bullet points about what happened. An attacker abused a video upload feature to upload a custom app to the server's deployment folder, from which the server was configured to automatically launch anything there. Ouch. It sounds like a lot of design improvements could have been made here to prevent that.
- 4. We updated our RSA SSH host key
The theme emerging this week is how to communicate security incidents, including communications that don't directly stem from incidents. I'll begrudgingly accept "we take security seriously" if an incident report goes into details that support that assertion. And I'll happily read "in an abundance of caution" write-ups that practice transparency about potential security issues.
In this case, their "RSA SSH private key was briefly exposed in a public GitHub repository" (ouch). This was human error, not a compromise, and they did the right things by immediately rotating keys and letting users know what was going on.
Although, in a few cases, SSH clients weren't very clear about what was going on. We'll ask John about that...
- 5. Bypassing CloudTrail in AWS Service Catalog, and Other Logging Research
We've been talking about logs recently, so this article grabbed my attention. It also fits our theme of incident write-ups. In this case, Datadog found a CloudTrail bypass, reported it to AWS, and AWS fixed it.
The broader picture is that that AWS customers wouldn't necessarily ever be aware of this. There's no equivalent CVE for tracking and measuring these kinds of incidents. On the other hand, AWS was able to resolve this without customer action needed and their audit of the impacted services indicated to malicious activity. So, it's nice that customers don't need to do anything here and none were apparently impacted by it.
- 6. We need a new way to measure AI security
There's more to AI security than prompt injections, and there's more than just adversarial attacks against uses like face recognition. This blog post and the paper it links proposes terminology and techniques for discussing risks and not only distinguishing between security and safety, but defining what safety should mean within the context of AI. It also explains why threat modeling most familiar to appsec may only be narrowly applicable and insufficient for reasoning about these systems.
- 7. Microsoft investigating reports of ‘aCropalypse’ image-crop vulnerability in Windows
We just covered this in episode 233, where it was disclosed as affecting Pixel's default image editor. Now it looks like Windows image editors join Android in keeping cropped out image areas within the image file's data.
- 1. Acropalypse hits Windows, too.
Seriously. How is it to write code to crop an image?
- 2. Unite for code quality!
Unfortunately, I'm partially posting this in jest. When I first saw the article, I was hoping to see mention of some developers union that was pushing companies to care more about code quality or something, but no...it's just a starry-eyed manifesto that probably won't get much traction.
Whoops. Grumpy John's coffee still hasn't kicked in on a Monday morning...
- 3. “Safe” npm
While I usually don't post links straight to vendors, Socket has released a "safe" version of npm which basically filters "npm install" requests through their filters to minimize the chance of installing "malware, typosquats, install scripts, protestware, telemetry" etc.
Why could Microsoft/Github/npm not come out with this???
- 4. Google Pixels Are Crashing After Watching This Alien Clip on YouTube