Another Ping of Death, Clever JSON Manipulation, iCloud Encryption, ChatGPT Threats – ASW #223

Highlighting this mostly as an item of curiosity and a vuln reminiscent of the ping of death in the late 90s.

As a quick recap, a buffer overflow can be triggered in FreeBSD's ping by a malformed packet. Crafting packets with incorrect length fields, conflicting flags, and overlapping fragments is a classic way to attack TCP/IP stacks. On the plus side, this looks like it's limited to a DoS since ping drops privileges once it opens a raw socket and it also executes within a sandboxed environment. Both of those are good practices for anything that operates on network data. On the other hand, it shows that C code remains plagued by memory safety issues.

This doesn't feel like the type of vuln that's going to be prevalent or a new class to worry about. But it's still an article worth reading! It reads like a walkthrough of a CTF challenge -- having a hunch about a vuln, running into obstacles while trying to exploit it, tweaking techniques based on technologies within the target, and eventually finding success. Look at it in terms of process and technique.

Apple adds encryption to iCloud backups. The guide shows why an apparently simple premise -- encrypt backups -- becomes complex in its edge cases, usability considerations, and policy impacts. This change expands the categories of data that Apple encrypts to user-managed keys, i.e. via end-to-end encryption. Apple enumerates the categories in this page.

Here's the Apple announcement

Everyone seems to be playing with chat AIs now. This article from NCC Group summarizes the prompt injection technique against such systems. The attack is essentially the mix of data and code that you'd see in SQL injection or command injection attacks. Only in this case you see more natural phrasing like, "Ignore the above directions and [do something unexpected]". It's a way of manipulating a model's instructions by interacting directly with the model rather than having it execute against the initial intent of a prompt. The article gives simple, clear examples.

The article includes several references (although several are just Twitter threads) that are worth reading should you find yourself intrigued by this class of attacks.

Bug bounty programs already struggle with low-quality reports and reports taken directly from scanners with no followup analysis by the submitter. Here's a case where a researcher apparently used ChatGPT to create a vuln report out of thin air. ChatGPT asserted there was a vuln in a smart contract, created a writeup, and requested a reward for identifying an authorization bypass.

But the vuln didn't exist, either in theory or practice. It just sounded reasonable to a non-expert. To quote from the article, “I was most surprised by the mixture of clear writing and logic, but built off an ignorance of 101-level coding basics."

The Cyber Grand Challenge demonstrated the potential for machine learning in vuln detection and exploitation. ChatGPT and its ilk may eventually improve on that, but it doesn't seem like something to worry about any time soon.

The prompt injection article this week covers one type of technical attack against AI implementations. Earlier this summer we also covered another paper from NCC Group about "Practical Attacks on Machine Learning Systems".

This article brings in the dimension of social impacts and the trust and safety issues that arise from AI-driven chatbots and image generators. Also read Wired's other article about Lensa from this week.

This paper goes in-depth into the how behind "Codex", a technology underpinning AI work like GitHub's Copilot. It also goes into analysis of the security implications of such technology, which share a similar threat landscape to Chat GPT.

A quote from this theoretical paper discussing AI and security is as follows; and describes best the essence of this article:

'Security is critical to ethical artificial intelligence not because it is a parallel property that we should seek alongside things like fairness, but because it is a prerequisite to it. Specifically, security is a prerequisite to ethical artificial intelligence because you cannot make meaningful guarantees about a system that you do not control.' -Jennifer Fernick

This article examines the basics of security risks posed by Chat GPT. The main conclusion is that while current AI cannot write sophisticated offensive security code, however, there may come a time in the future where it can. Furthermore, the biggest risk of current code writing tools and security isn't so much exploitation of existing software, it is that much of the code that it does churn out is insecure. https://www.cyberscoop.com/chatgpt-ai-malware/

A more lighthearted article on getting ChatGPT to provide all sorts of instructions for nefarious deeds.