Malware Development Series: Part 1

⚠️ Ethical Use Disclaimer: This content is for educational and ethical security research purposes only. Knowledge of malware development should only be used for proper penetration testing, red team operations and defensive security research. Unauthorized use will be subject to criminal prosecution.

Introduction

Malware development has evolved into essential new skills for modern security professionals, especially for those conducting offensive security assessments. This guide walks through the basic concepts, tools, and methods to enable understanding of malware development for ethical security research.

What is Malware and Why Learn Development?

Malware refers to software that has been deliberately created to compromise systems or steal sensitive information among other actions. While the term "malware" is often associated with criminal or questionable activity, malware development can be done for legitimate academic research by ethical hackers and penetration testers performing authorized security assessments such as red teaming.

Why Security Professionals Need Malware Development Skills

When conducting offensive assessments, security teams usually have three options.

• Open-Source Tools (OSTs), which are generally signatured by security vendors, highly detectable in a mature environment.

• Commercial Tools, which are more evasive, costly and still ultimately known to the defenders.

• Custom Tools, which are most evasive because they have not been analyzed or signatured by security vendors.

Developing custom malware gives offensive security teams a distinct advantage in representing realistic assessments of an organization's defensive capabilities.

Programming Language Considerations

High-Level vs Low-Level Languages

There are some considerations when selecting a programming language for malware development.

• High-Level Languages like, Python, C#, and PowerShell are more abstracted from the operating system , often have runtime dependencies on target systems, easier to develop but easier to detect and analyze , with our automated development , have a larger file size.

• Low-Level Languages like, C, C++, and Assembly interact directly with the operating system, offer much more control over the resources and behavior , harder to reverse engineer , smaller executables (size and memory), more efficient, and no external run-time dependencies.

Diagram: Comparison of high-level vs low-level languages in malware development context

Why C Dominates Windows Malware Development

For a number of reasons, the C programming language has become the language of choice for Windows-based malware development. Its ability to access Windows API directly without abstraction layers, low dependency with no external runtimes (it only relies on Windows), reverse engineering resistance due to the fact compiled C code is much harder to reverse engineer than .NET languages, the power of memory management (which is an important technique for evasion), and small executable sizes, which are a positive aspect for delivery and stealth.

The Malware Development Life Cycle (MDLC)

Successful malware development follows a structured process similar to traditional software development:

1. Development Phase

Begin development or refinements to malware functionality with the most attention on developing the core functionality of the payload, evasion methods, communication methods, and persistence methods.

2. Testing Phase

Conduct a full suite of testing to discover any issues and bugs by testing functionality, testing error handling, testing edge cases, and testing code stability.

3. Offline AV/EDR Testing

Test with antivirus solutions without internet connectivity through isolated testing environments and disable the cloud based detection features, and also try to detect patterns and signatures.

4. Online AV/EDR Testing

Test with Internet access, or via services like VirusTotal to evaluate cloud-based detection engines and evaluate behavioral analysis capabilities. Caution: This may result in samples being submitted to vendors.

5. IoC Analysis

From the perspective of a defender, analyze the malware by identifying possible indicators of compromise, considering detection capabilities, documenting behaviour signatures, and reflecting on possible improvements to evasion techniques.

The cycle then repeats, continuously improving the malware's effectiveness and evasion capabilities.

Diagram: Visual representation of the 5-phase MDLC showing the iterative improvement process

Essential Development and Analysis Tools

Development Environment

Visual Studio

• Primary IDE for C/C++ malware development
• Integrated debugging and compilation tools
• Extensive Windows SDK integration
• Essential for building Windows-native applications

Reverse Engineering and Analysis Tools

x64dbg is an open-source debugger for Windows binaries, and is a crucial tool for dynamic analysis and understanding malware behavior. x64dbg provides four main analysis windows: "for viewing assembly instructions, memory dumps, for the raw contents of memory, registers, for the values of CPU registers, and Stack, for the contents of the stack frame.

In addition, PE-Bear is a Portable Executable (PE) file analysis tool that offers static analysis capabilities, import/export table examiner, and a PE header visualizer.

Process Hacker 2 is a powerful tool for advanced process monitoring and manipulation, DLL analysis, memory region analysis, providing more detailed information than the Task Manager.

Msfvenom is a Metasploit payload generation tool that can create multiple payload formats and is useful for testing and proof-of-concept development.

Windows Architecture Fundamentals

Understanding Windows architecture is crucial for effective malware development.

User Mode vs Kernel Mode

Windows operates in two privilege levels:

• User Mode, where user applications and processes execute. Applications and processes reside in a "protected" memory space, which has limited access to system resources.
• Kernel Mode, where the OS components execute. Kernel components have complete unrestricted access to system resources in order to support critical system operations and functions.

Windows API Function Call Flow

When an application needs to perform system operations, it follows this call chain:

1. User Application calls a Windows API function
2. Subsystem DLL (e.g., kernel32.dll) processes the request
3. Ntdll.dll provides the transition to kernel mode via Native API
4. System Call transfers execution to kernel mode
5. Kernel executes the requested operation

Example: File Creation Flow

Application: CreateFileW()
    ↓
Kernel32.dll: CreateFileW() wrapper
    ↓
Ntdll.dll: NtCreateFile()
    ↓
Syscall instruction (syscall/sysenter)
    ↓
Kernel: NtCreateFile() implementation

Practical Debugging Demonstration

Let's trace this call flow using WinDbg with a simple C program:

Test Program (main.c):

WinDbg Setup Commands:

.symfix
.reload

Essential WinDbg Commands for This Analysis:

• g (Go): Continues execution until the next breakpoint is hit
• k (Stack): Displays the current call stack showing the function call chain
• r (Registers): Shows current CPU register values
• uf (Unassemble Function): Disassembles a complete function to show assembly instructions

Step 1: Breakpoint at Application Layer

bp kernel32!CreateFileW

This captures the initial Windows API call from your application.

Step 2: Breakpoint at Native API Layer

bp ntdll!NtCreateFile

This is where the high-level API transitions to the Native API layer.

Step 3: Examine the Syscall Instruction First, disassemble the NtCreateFile function:

uf ntdll!NtCreateFile

This reveals the complete syscall preparation:

Step 4: Breakpoint on Syscall Transition

bp 00007ffc`e8fdca62

This captures the exact moment of user-mode to kernel-mode transition at the syscall instruction.

Debugging Results:

Breakpoint 0: Application calls CreateFileW - shows the initial API entry point

Breakpoint 1: Native API layer - shows the call stack traversal through KERNELBASE

Breakpoint 2: Syscall instruction - the actual user-to-kernel mode transition

Key Observations from the Debug Session:

1. Modern Windows Architecture: The call actually goes through KERNELBASE.dll rather than directly from KERNEL32.dll
2. Application Compatibility: apphelp.dll hooks are visible in the call stack, showing Windows compatibility layers
3. System Call Number: 0x55 (85 decimal) is the syscall number for NtCreateFile on this Windows version. This syscall number is directly used by advanced malware to bypass AV/EDR solutions by invoking syscalls directly rather than going through the normal API layers.
4. Register Usage: r10 preserves the first parameter (rcx) before the syscall instruction

Interactive API Call Flow Visualization

#include <windows.h>

HANDLE hFile = CreateFileW(
    L"C:\\temp\\test.txt",
    GENERIC_WRITE,
    0,
    NULL,
    CREATE_ALWAYS,
    FILE_ATTRIBUTE_NORMAL,
    NULL
);

Direct Native API Invocation

Advanced malware frequently conducts its API calls directly to Native API functions, beyond the Windows API layer it also providing more control over the system. However there are also some disadvantages: the APIs are undocumented and may change, they are more complicated to implement and require a deeper understanding of Windows internals.

Security and Evasion Considerations

Modern malware development must focus heavily on evasion. Host-Based Security Evasion includes Antivirus (AV) bypass techniques, Endpoint Detection and Response (EDR) evasion, behavioral analysis avoidance, and signature-based detection circumvention.

Operational Security (OpSec) involves avoiding "malware-like" behaviors that trigger detection, implementing legitimate-looking functionality, using living-off-the-land techniques, and minimizing forensic artifacts.

Testing and Validation

Offline Testing Best practices involve isolating VMs, disabling automatic sample submission, testing against different security products, and documenting detection patterns in order to come up with improvements.

Online Testing Considerations include understanding the risk involved in submitting samples. You need to be aware that they may be uploaded and sometimes rendering payloads obsolete.

Conclusion

Malware Development for ethical security research takes a lot of effort and thorough understanding of the Windows architecture, solid understanding of programming fundamentals as well as current evasion techniques. These skills are necessary in order to develop custom tools to conduct more effective penetration tests or even understand threat actor techniques, etc..

The field continues to evolve rapidly as both offensive and defensive security move forward. Security professionals working in this area need to constantly stay up to date with the latest techniques, tools, and countermeasures.

Additional Resources and Next Steps

Windows Development Resources

• Microsoft Windows SDK Documentation
• Windows API Reference
• Windows Internals Book Series
• MSDN Win32 Programming Guide
• Windows Driver Kit (WDK) Documentation
• Debugging Tools for Windows

Next in Series: Part 2 will cover Windows Memory Management, Introduction to the Windows API, and Portable Executable Format and Dynamic-Link Library (DLL) structures.

Important Legal and Ethical Considerations

Only use knowledge for authorized testing, obtain proper written authorization, follow responsible disclosure practices, maintain strict operational security, and document all activities for legal protection.

Remember: The goal of learning malware development is to improve security, not to cause harm. Always ensure your activities are legal, authorized, and conducted within appropriate boundaries.