A group of MITRE researchers have pitched a new standard for defining security threats from AI systems.
Dubbed OCCULT, the new framework would provide cybersecurity professionals and researchers with a set of criteria for evaluating how large language models (LLMs) would be able to conduct cyberattacks and the risk that network defenders should assess to such threats.
“Offensive Cyber Operations (OCO) have often historically required highly educated computer operators, multidisciplinary teams, mature targeting and development processes, and a heavily resourced sponsoring organization to viably execute at scale and to a mission effect,” the researchers wrote.
“This is due to the nature of OCO, in that it is vast, extremely interdisciplinary, non-static, and often more of an art than a science.”
To fill in that gray area, the MITRE team suggests a framework that will help security professionals evaluate AI-based threat actors on a uniform basis and apply a set method of scoring and prioritizing attacks.
“To mitigate the potential risk that autonomous and even semi-autonomous AI-enabled OCO systems could pose, we must be able to evaluate the true capabilities of any emerging model rigorously and swiftly,” the researchers said.
“As with any field with the depth and breadth of cyber security, simply testing knowledge recall or memory is insufficient. Instead, the application of OCO capabilities requires knowledge and domain models, information synthesis, perception of environment/state, action and solution spaces, and use of tools and intelligence generalization.”
The emergence of AI systems and LLMs has created a new frontier for security vendors who now have to face the threat of automated systems that can streamline the process of scanning networks for vulnerabilities as well as send social engineering messages aimed at harvesting user credentials.
In short, the MITRE crew is suggesting a method that would allow network defenders to assess the threat posed by AI systems and their operators.
To do that, the researchers pre-planned several scenarios ranging from situations where an AI would assist in the research and presentation of social engineering attacks that would then be executed by a human threat actor, to fully automated attacks that would be carried out by the AI deployment.
“Our research team has created just such a light methodology and evaluation framework (known as OCCULT) for designing rigorous and repeatable evaluations that can quantify the unique, plausible cyber security risks associated with an LLM used in OCO,” the researchers explained.
The MITRE team is putting forward plans for a framework to assess and attribute risks.
“These tenets address the current gaps we see in the literature and open-source work regarding the evaluation of LLMs for offensive cyber capabilities, which requires measuring performance across different cyber tools, environments, and use-cases to encompass the breadth and depth of the offensive cyber landscape,” the researchers concluded.
