DBIS

Short Description:
This thesis aims to develop a framework for evaluating how accurately Large Language Model (LLM) pipelines convert legacy community playbooks into CACAO-based cybersecurity playbooks. The main focus is on ensuring semantic fidelity: transforming both original and translated workflows into a unified BPMN (or less preferred Petri-nets) representation, then applying process-mining metrics to compute similarity scores and generate detailed discrepancy reports and feed back to the LLM for automatic refinements. By capturing both control-flow and data-flow within BPMN, the framework intends to quantify and refine LLM translation accuracy, enabling standardized cyber incident response procedures without compromising workflow logic. The research emphasizes the use of process mining techniques and similarity checks to evaluate and optimize playbook workflows. Moving beyond traditional graph edit distance, the study seeks to identify and apply superior metrics for process similarity analysis, providing enhanced insights into the workflows and ensuring semantic validation of translated playbooks. The goal is to introduce a novel evaluation framework for the LLM-generated workflows of playbooks.

Relevant links and literatures:

• CACAO Security Playbooks Version 2.0: https://docs.oasis-open.org/cacao/security-playbooks/v2.0/cs01/security-playbooks-v2.0-cs01.html
• Process Modeling With Large Language Models: https://arxiv.org/pdf/2403.07541.pdf
• A Method for Extracting BPMN Models from Textual Descriptions Using Natural Language Processing: https://zir.nsk.hr/islandora/object/unipu:8207/datastream/PDF/download
• Example community playbooks repositories can be input for our approach:
  • https://github.com/luduslibrum/awesome-playbooks (probobly best one with 1300+ playbooks)
  • https://github.com/phantomcyber/playbooks
  • https://gitlab.com/syntax-ir/playbooks

Research Questions:

• How can diverse cybersecurity playbooks be transformed into a standardized BPMN workflow while preserving their essential procedural logic and data dependencies?
• Which process mining and similarity techniques effectively evaluate the fidelity of LLM-translated playbooks, ensuring minimal loss or alteration of critical workflow elements?
• In what ways can detected discrepancies be utilized to refine translation strategies and enhancing the performance and reliability of future LLM-based playbook conversions?