<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">This brand new paper may be of interest to our AI+Cybersecurity faculty:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities<o:p></o:p></b></p>
<p class="MsoNormal">The prospect of artificial intelligence (AI) competing in the adversarial landscape of cyber<o:p></o:p></p>
<p class="MsoNormal">security has long been considered one of the most impactful, challenging, and potentially<o:p></o:p></p>
<p class="MsoNormal">dangerous applications of AI. Here, we demonstrate a new approach to assessing AI’s progress<o:p></o:p></p>
<p class="MsoNormal">towards enabling and scaling real-world offensive cyber operations (OCO) tactics in use by<o:p></o:p></p>
<p class="MsoNormal">modern threat actors. We detail OCCULT, a lightweight operational evaluation framework<o:p></o:p></p>
<p class="MsoNormal">that allows cyber security experts to contribute to rigorous and repeatable measurement of<o:p></o:p></p>
<p class="MsoNormal">the plausible cyber security risks associated with any given large language model (LLM) or AI<o:p></o:p></p>
<p class="MsoNormal">employed for OCO. We also prototype and evaluate three very different OCO benchmarks for<o:p></o:p></p>
<p class="MsoNormal">LLMs that demonstrate our approach and serve as examples for building benchmarks under the<o:p></o:p></p>
<p class="MsoNormal">OCCULT framework. Finally, we provide preliminary evaluation results to demonstrate how<o:p></o:p></p>
<p class="MsoNormal">this framework allows us to move beyond traditional all-or-nothing tests, such as those crafted<o:p></o:p></p>
<p class="MsoNormal">from educational exercises like capture-the-flag environments, to contextualize our indicators<o:p></o:p></p>
<p class="MsoNormal">and warnings in true cyber threat scenarios that present risks to modern infrastructure. We<o:p></o:p></p>
<p class="MsoNormal">find that there has been significant recent advancement in the risks of AI being used to<o:p></o:p></p>
<p class="MsoNormal">scale realistic cyber threats. For the first time, we find a model (DeepSeek-R1) is capable<o:p></o:p></p>
<p class="MsoNormal">of correctly answering over 90% of challenging offensive cyber knowledge tests in our Threat<o:p></o:p></p>
<p class="MsoNormal">Actor Competency Test for LLMs (TACTL) multiple-choice benchmarks. We also show how<o:p></o:p></p>
<p class="MsoNormal">Meta’s Llama and Mistral’s Mixtral model families show marked performance improvements<o:p></o:p></p>
<p class="MsoNormal">over earlier models against our benchmarks where LLMs act as offensive agents in MITRE’s<o:p></o:p></p>
<p class="MsoNormal">high-fidelity offensive and defensive cyber operations simulation environment, CyberLayer.<o:p></o:p></p>
<p class="MsoNormal"><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Farxiv.org%2Fpdf%2F2502.15797&data=05%7C02%7Ccaice-csse%40eng.auburn.edu%7C204225d4e01b4a08871a08dd567bfa74%7Cccb6deedbd294b388979d72780f62d3b%7C0%7C0%7C638761812657648992%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=v2wgBSVavhJ3AfKRX1xU5EwCN3idByUir9tR8JXeewE%3D&reserved=0" originalsrc="https://arxiv.org/pdf/2502.15797">https://arxiv.org/pdf/2502.15797</a>
<o:p></o:p></p>
</div>
</body>
</html>