Malware Development Series: Part 2

⚠️ Ethical Use Disclaimer: This content is for educational and ethical security research purposes only. Knowledge of malware development should only be used for proper penetration testing, red team operations and defensive security research. Unauthorized use will be subject to criminal prosecution.

Introduction

Part 2 continues from the materials presented in Part 1 about Windows architecture and the fundamentals of Windows development. In this chapter, we will develop a more sophisticated understanding of the technical components that every security professional must understand: Windows memory management, Windows API, PE file layouts, and DLLs (Dynamically-Linked Libraries).

These concepts form the basis for how advanced malware is developed, whether you work on offensive security teams that create custom tools for red teams, or whether you are on a defensive team and need to understand the techniques of the threat actors you are facing.

Windows Memory Management

Understanding how Windows handles memory is crucial for building advanced malware and evasion techniques. Modern memory management provides both opportunities and challenges for security professionals.

Virtual Memory and Paging

In modern operating systems, memory does not map directly to physical memory (RAM). Instead, a process will use virtual memory addresses, which in turn are mapped to physical memory . This approach provides the following important advantages:

• Memory Savings: Physical memory is saved through "pages" on disk.
• Process isolation :Each process has its own virtual address space, which ensures processes cannot accidentally (or maliciously) access each other’s memory.
• Memory sharing: Virtual memory allows different processes to map the same physical memory (e.g., shared libraries) to different virtual addresses, enabling efficient inter-process communication or shared code.
• Security: It is not possible for different processes to directly access each other's memory

Virtual memory allows for this separation using memory paging which treats memory as 4KB pages, , though many support larger pages (e.g., 2MB or 1GB "huge pages") for performance optimization. This structure allows the operating system to better manage memory, and in particular allocate memory pages.

Page States

Pages within a process's virtual address space can exist in one of three states:

Free Pages

Free pages are like completely empty rooms that haven’t been claimed or prepared for anything. They’re not connected to your stuff, and you’re not allowed to go in. If your program tries to access them the system throws an error (an access violation). But these pages are ready to be used in the future when needed.

Reserved Pages

Reserved pages are more like rooms you’ve marked off for future use: “This space is mine, and I might set it up later.” No one else can use them, but they’re still empty. You haven’t put anything in them yet, and you can’t use them until you commit to setting them up; that is, until your program says, “Okay, I’m ready to use this space now.”

Committed Pages

Committed pages are the rooms you’ve actually furnished and started using. They take up real space, either in your computer’s RAM or on disk, and your program can read from or write to them, depending on what kind of access you’ve requested. These pages don’t necessarily get loaded into physical memory right away. Instead, they’re brought in when your program first touches them. Once you’re done with them, or when your program closes, the system cleans them up.

Page Protection Options

Once pages are committed, they require protection settings that define access permissions:

• PAGE_NOACCESS: Disables all access and any attempt results in access violation
• PAGE_EXECUTE_READWRITE: Enables read, write, and execute permissions
  • ⚠️ Security Note: This is highly discouraged and often flagged as suspicious since it's uncommon for memory to be both writable and executable simultaneously
• PAGE_READONLY: Read-only access write attempts cause access violations
• PAGE_EXECUTE_READ: Execute and read permissions - commonly used for code sections
• PAGE_READWRITE: Read and write permissions - typically used for data sections

Modern Memory Protections

Modern versions of Windows include smart defenses to make it harder for attackers, or malware, to mess with memory. Two of the most known ones are Data Execution Prevention (DEP) and Address Space Layout Randomization (ASLR).

Data Execution Prevention (DEP)

Data Execution Prevention (DEP) is like posting a sign "No Running" in areas of memory where it only wants to preserve memory. If something is in fact trying to execute code in an area that is set to preserve data, DEP will step in and put an end to the offending instruction to prevent anything more sinister from occurring.

Address Space Layout Randomization (ASLR)

Address Space Layout Randomization (ASLR) operates in a similar fashion to moving the furniture in a house every time someone walks in. It randomly changes where programs, libraries, and important parts of memory are located, rendering the attacker unable to know where to strike. This causes attackers to have a harder time successfully executing exploits as if you don't know where something is, then it's much harder to attack it.

Memory Architecture Differences

On 32-bit (x86) systems, processes have a maximum of 4GB of virtual memory space, which sounds like a lot, but becomes a limiting factor for modern applications very quickly. 64-bit (x64) systems increase that tremendously. In theory, 64 bit architecture allows use of 16 exabytes of memory, but in practice, user-mode applications in most operating systems (including Windows) only allow use of up to 128 terabytes of virtual address space.

Practical Memory Allocation Examples

Understanding memory allocation is fundamental for malware development. Here are the primary methods:

Method 1: Using malloc()

Method 2: Using HeapAlloc()

Method 3: Using LocalAlloc()

Memory Initialization Considerations

When memory is allocated, it may contain random data. Some allocation functions provide options to zero out memory:

Proper Memory Cleanup

Always deallocate memory to prevent memory leaks:

• malloc() requires free()
• HeapAlloc() requires HeapFree()
• LocalAlloc() requires LocalFree()

Interactive Memory Management Visualization

Introduction to the Windows API

The Windows API provides the interface between applications and the Windows operating system. For malware developers, mastering the Windows API is essential for creating effective and evasive tools.

Windows Data Types

The Windows API introduces many specialized data types beyond standard C types:

Essential Data Types

String Data Types

Pointer Arithmetic Type

ANSI vs Unicode Functions

Most Windows API functions have two versions:

• "A" suffix: ANSI functions (8-bit characters)
• "W" suffix: Unicode/Wide functions (16-bit characters)

Parameter Directions

Windows APIs use directional parameter annotations:

• IN parameters: Input data passed to the function
• OUT parameters: Output data returned from the function (often via pointers)
• OPTIONAL: Parameter may be NULL or omitted

Practical Windows API Example

Let's examine the complete process of using CreateFileW:

Error Handling

Windows API Error Handling

Windows APIs return specific values on failure and use GetLastError() for detailed error information:

if (hFile == INVALID_HANDLE_VALUE) {
    DWORD error = GetLastError();
    printf("[-] CreateFileW failed with error: %lu\n", error);
    
    // Common error codes:
    // 5   - ERROR_ACCESS_DENIED
    // 2   - ERROR_FILE_NOT_FOUND  
    // 87  - ERROR_INVALID_PARAMETER
}

Native API Error Handling

Native APIs (NTAPI functions) return NTSTATUS values directly:

NTSTATUS status = NtCreateFile(...);
if (status != STATUS_SUCCESS) {
    printf("[!] NtCreateFile failed with status: 0x%08X\n", status);
}

// Alternative using NT_SUCCESS macro
if (!NT_SUCCESS(status)) {
    printf("[!] Function failed with status: 0x%08X\n", status);
}

Portable Executable (PE) Format

The Portable Executable format is the standard for executables on Windows. Understanding PE structure is crucial for malware development, analysis, and evasion techniques.

PE File Types

• .exe - Executable files
• .dll - Dynamic Link Libraries
• .sys - System drivers
• .scr - Screen savers
• .cpl - Control Panel applications

PE Structure Overview

Every PE file follows a consistent structure with multiple headers providing metadata about the executable:

[DOS Header] → [DOS Stub] → [NT Header] → [Section Headers] → [Sections]

DOS Header (IMAGE_DOS_HEADER)

The first header in every PE file, always starting with the "MZ" signature:

Key Members:

• e_magic: Fixed value 0x5A4D ("MZ") - PE file signature
• e_lfanew: Offset to NT Header (critical for parsing)

DOS Stub

A small program that displays "This program cannot be run in DOS mode" when executed in DOS. This message can be customized by developers and is sometimes used for anti-analysis techniques. Malware authors often modify the DOS stub to include misleading information, fake error messages, or even decoy code to confuse automated analysis tools and reverse engineers.

NT Header (IMAGE_NT_HEADERS)

The core header containing PE signature and two sub-headers:

File Header (IMAGE_FILE_HEADER)

Important Members:

• NumberOfSections: Count of sections in the PE
• TimeDateStamp: Unix timestamp (seconds since Jan 1, 1970 UTC) - useful for determining compilation time
• Characteristics: Flags indicating file type and properties
  • IMAGE_FILE_EXECUTABLE_IMAGE (0x0002): File is executable
  • IMAGE_FILE_DLL (0x2000): File is a DLL, not an executable
• SizeOfOptionalHeader: Size of following optional header

Optional Header (IMAGE_OPTIONAL_HEADER)

Despite the name, this header is essential for executable files. Note that there are two versions:

• IMAGE_OPTIONAL_HEADER32: For 32-bit executables
• IMAGE_OPTIONAL_HEADER64: For 64-bit executables (shown below)

Critical Members:

• AddressOfEntryPoint: RVA where execution begins (relative to ImageBase, often main function)
• ImageBase: Preferred memory load address (affected by ASLR)
  • 32-bit: Typically 0x00400000 for EXEs, 0x10000000 for DLLs
  • 64-bit: Typically 0x0000000140000000 for EXEs
• SizeOfImage: Total memory footprint when loaded
• DataDirectory: Array of important PE structures (detailed below)

Data Directory

The Data Directory is an array of IMAGE_DATA_DIRECTORY structures pointing to important PE components:

typedef struct _IMAGE_DATA_DIRECTORY {
    DWORD   VirtualAddress;    // RVA of the data
    DWORD   Size;             // Size of the data
} IMAGE_DATA_DIRECTORY, *PIMAGE_DATA_DIRECTORY;

Important Data Directory Entries:

• Index 0: Export Directory - Functions exported by this PE (for DLLs)
• Index 1: Import Directory - Functions imported from other PEs
• Index 2: Resource Directory - Icons, strings, version info, etc.
• Index 3: Exception Directory - Exception handling information (x64)
• Index 5: Base Relocation Table - Address fixups for ASLR
• Index 6: Debug Directory - Debug information
• Index 9: Thread Local Storage (TLS) Directory
• Index 12: Import Address Table (IAT) - Runtime function addresses
• Index 14: Delay Import Directory - Delayed loading imports

These directories are critical for malware analysis as they reveal:

• Import Table: What APIs the malware uses
• Export Table: What functions it provides (if it's a DLL)
• Resource Section: Embedded files, configuration data
• Relocation Table: Whether ASLR can be applied

PE Sections

PE sections contain the actual code and data:

Common Sections

• .text: Executable code (your written code)
• .data: Initialized global/static variables
• .rdata: Read-only data (constants, import tables)
• .idata: Import tables and related structures
• .reloc: Relocation information for ASLR
• .rsrc: Resources (icons, bitmaps, strings)

Section Headers

Each section has an IMAGE_SECTION_HEADER describing its properties:

typedef struct _IMAGE_SECTION_HEADER {
    BYTE  Name[8];              // Section name (null-padded)
    DWORD VirtualSize;          // Size in memory
    DWORD VirtualAddress;       // RVA of section in memory
    DWORD SizeOfRawData;        // Size on disk
    DWORD PointerToRawData;     // File offset of section data
    // ... relocation fields ...
    DWORD Characteristics;      // Section permissions/characteristics
} IMAGE_SECTION_HEADER, *PIMAGE_SECTION_HEADER;

Dynamic-Link Library (DLL)

In Windows, a Dynamic-Link Library (DLL) is a unique kind of file that contains the reusable code (and other resources). Developers can separate common or specialized tasks into different DLLs instead of embedding functionality directly into an application. This results in more modular software that is typically better organized, more efficient, and easier to maintain.

What Makes DLLs Special

Dynamic link libraries (DLLs) allow multiple programs use the same code, thus, improving memory usage since a DLL is only loaded once into memory and it can be shared among multiple processes. Some system DLLs like kernel32.dll and user32.dll are commonly referenced by many applications in order to use core functionality.

DLLs also promote modularity in that feature development can be done separately from the main application and developers can easily patch new features without having to patch the whole application.

DLLs can also be loaded at runtime with Windows-specific APIs which also enables more flexibility. There can even be functionality that can make the malware run stealthily. Functionality that can hide or "stealth" calling a function can be programmed using things like LoadLibrary() and GetProcAddress().

System-Wide DLL Base Addresses

Windows optimizes memory usage by loading common DLLs at the same virtual address across all processes:

kernel32.dll:    0x7FFAB6420000  
ntdll.dll:       0x7FFAB7FE0000  
user32.dll:      0x7FFAB6980000  

DLL Entry Point

DLLs can specify an entry point function called during specific events:

BOOL APIENTRY DllMain(
    HMODULE hModule,           // Handle to DLL module
    DWORD ul_reason_for_call,  // Reason for calling
    LPVOID lpReserved          // Reserved parameter
) {
    switch (ul_reason_for_call) {
        case DLL_PROCESS_ATTACH:
            // Process is loading the DLL
            break;
        case DLL_THREAD_ATTACH:
            // Process is creating a new thread
            break;
        case DLL_THREAD_DETACH:
            // Thread exits normally
            break;
        case DLL_PROCESS_DETACH:
            // Process unloads the DLL
            break;
    }
    return TRUE;
}

Exporting Functions from DLLs

Functions must be explicitly exported to be available to other modules:

// Export function definition
extern __declspec(dllexport) void HelloWorld() {
    MessageBoxA(NULL, "Hello from DLL!", "DLL Function", MB_OK);
}

Dynamic Linking

Loading and using DLL functions at runtime provides flexibility and evasion opportunities:

Step 1: Load the DLL

// Load DLL into process memory
HMODULE hModule = LoadLibraryA("user32.dll");
if (hModule == NULL) {
    printf("[-] Failed to load library\n");
    return -1;
}

Step 2: Get Function Address

// Retrieve function address
PVOID pMessageBoxA = GetProcAddress(hModule, "MessageBoxA");
if (pMessageBoxA == NULL) {
    printf("[-] Failed to get function address\n");
    FreeLibrary(hModule);
    return -1;
}

Step 3: Create Function Pointer

// Define function pointer type
typedef int (WINAPI* fnMessageBoxA)(
    HWND   hWnd,
    LPCSTR lpText,
    LPCSTR lpCaption,
    UINT   uType
);

// Cast address to function pointer
fnMessageBoxA pMessageBox = (fnMessageBoxA)pMessageBoxA;

Step 4: Call the Function

// Invoke the function
pMessageBox(NULL, "Dynamic Loading Success!", "Malware Dev", MB_OK);

// Cleanup
FreeLibrary(hModule);

Complete Dynamic Loading Example

#include <windows.h>
#include <stdio.h>

// Function pointer type definition
typedef int (WINAPI* fnMessageBoxA)(HWND, LPCSTR, LPCSTR, UINT);

int main() {
    // Load user32.dll
    HMODULE hUser32 = LoadLibraryA("user32.dll");
    if (hUser32 == NULL) {
        printf("[-] Failed to load user32.dll\n");
        return -1;
    }
    
    // Get MessageBoxA address
    fnMessageBoxA pMessageBoxA = (fnMessageBoxA)GetProcAddress(hUser32, "MessageBoxA");
    if (pMessageBoxA == NULL) {
        printf("[-] Failed to get MessageBoxA address\n");
        FreeLibrary(hUser32);
        return -1;
    }
    
    // Call the function
    pMessageBoxA(NULL, "Dynamic Loading Example", "Success", MB_OK);
    
    // Cleanup
    FreeLibrary(hUser32);
    return 0;
}

Rundll32.exe - Built-in DLL Execution

Windows provides rundll32.exe for executing DLL functions directly:

# Lock the workstation using User32.dll
rundll32.exe user32.dll,LockWorkStation

# Open Control Panel applets
rundll32.exe shell32.dll,Control_RunDLL desk.cpl

Because rundll32.exe is a secure system binary, it is a target used in Living-off-the-Land (LotL) attacks. Malware authors utilize these trusted images to obfuscate their systems activities, hiding functionality and placing their code in a signed Windows process to remain undetected. This also gets around application whitelisting as security and other controls are less likely to block proof binary supplied functionality, versus finding the malicious or unwanted activity of a randll32.exe function.

Attackers often run rundll32.exe to run a .dll themselves, avoiding dropping a separate loader or executable, which keeps their activities quieter and harder to forensically track.

Conclusion

Part 2 unearthed important Windows internals; memory management, the Windows API, PE file structure, and DLL behavior. These components are fundamental in both developing advanced malware, and understanding how it works.

Whether writing tooling, or reversing threats, it is imperative to have a solid understanding of how Windows handles code and memory. In a field that is constantly changing, an understanding of these components are key in staying ahead.

Next in Series: Part 3 will cover detection mechanisms like signature and heuristic analysis, behavior-based detection, API hooking, and IAT checking. It will also introduce Windows processes, including threads, memory types, and the Process Environment Block (PEB).

Additional Resources

• Windows Memory Management Documentation
• PE Format Specification
• Windows API Reference
• 0xRick's PE Internals Series
• Malware Analysis Tools and Techniques

Important Legal and Ethical Considerations

Only use knowledge for authorized testing, obtain proper written authorization, follow responsible disclosure practices, maintain strict operational security, and document all activities for legal protection.

Remember: The goal of learning malware development is to improve security, not to cause harm. Always ensure your activities are legal, authorized, and conducted within appropriate boundaries.