Enhancing Malware Detection with AI-Assisted Reverse Engineering

Abstract

The cybersecurity industry faces a critical challenge due to the rapid evolution of malware, necessitating the development
of sophisticated detection methods that can generalize across a broad range of threats. This paper investigates the utilization
of the bias–variance tradeoff in AI-assisted reverse engineering (AIARE) to enhance the functionality of malware and threat
detection systems. By employing the Random Forest algorithm, an ensemble learning approach, we optimize the classification
of malicious software by addressing the inherent tradeoff between variance and bias. The proposed method uses features extracted from reverse-engineered code, including opcode sequences and behavioral signatures, to create decision trees that balance
model complexity and generalization optimally. To mitigate overfitting and maintain sensitivity to various malware types, critical hyperparameters such as the number of trees, tree depth, and the number of features per split are adjusted. The experimental results demonstrate that this technique improves the accuracy, robustness, and adaptability of detection systems against new threats, making it a valuable tool for contemporary cybersecurity solutions. This work strengthens defenses against increasingly sophisticated
cyber-attacks and advances AI-driven methods for threat intelligence.

Introduction

AI-assisted reverse engineering (AIARE) is a field of computer science that utilizes artificial intelligence (AI), specifically machine learning (ML) strategies, to automate and enhance the reverse engineering process. Reverse engineering involves dissecting a product, system, or process to understand its structure, design, and functionality. Since its introduction in the early 21st century, AIARE has made significant strides, particularly since the mid-2010s.

Reverse engineering traditionally requires significant....